Skip to contentSkip to conversation

Problem with RAID-5 not adding disks

DISCUSSION Theophanis Kontogiannis 3 participants 8 messages Jul 22, 2014 – Jul 24, 2014 1 message not in the archive MD RAID

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Theophanis Kontogiannis

author
#1 lore raw

Dear List.

Hello.

I have the following problem with my CEntOS 6.5 RAID-5

Array of five disks.

At some point in time I added the fifth disk, however I do not remember if I added it as a spare or to go up to RAID-6.

The power and faulty UPS caught me so my RAID-5 has failed (alongside with the on board controller :) )

After replacing the MB, I am in the following situation:

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 21 18:06:31 2014
Raid Level : raid5
Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
Raid Devices : 5
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sat Jul 12 22:11:39 2014
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : tweety.example.com:0 (local to host
tweety.example.com)
UUID : 953836cd:23314476:5db06922:c886893d
Events : 16855

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 32 1 active sync /dev/sdc
2 0 0 2 removed
4 8 64 3 active sync /dev/sde
4 0 0 4 removed

Every effort to add at least one more disk ends up in error:

[root ~]# mdadm /dev/md0 --re-add /dev/sdd
mdadm: --re-add for /dev/sdd to /dev/md0 is not possible

I also made sure that the devices are in the correct physical order.

    mdadm --examine /dev/sd[b-f]


        /dev/sdb:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x2
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953260911 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
        Recovery Offset : 0 sectors
                  State : active
            Device UUID : 7117d844:783abda5:093ee4d9:ba0ac2f0

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : 31569c40 - correct
                 Events : 16845

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 0
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sdc:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 18bf4347:08c262ec:694eba7f:eb8e6b26

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : 55e8716a - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 1
           Array State : .AAAA ('A' == active, '.' == missing)
        /dev/sdd:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 4f77c847:2f567632:66bb5600:9fb2eeba

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : b19f455b - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 2
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sde:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 51b43b6e:7bb8a070:f6375540:28e0b75c

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : f80197e3 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 3
           Array State : .A.AA ('A' == active, '.' == missing)
        /dev/sdf:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 9f7b1355:936e12f2:813c83a0:13854a6b

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : cf0bbec1 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 4
           Array State : .A.AA ('A' == active, '.' == missing)

Any chance to revive this array?

Signature · 12 lines
Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#2 lore raw

mdadm --stop /dev/mdXX stop
mdadm --assemble --force /dev/mdXX <list all devices good or bad that were previously in the array>

The assemble with force should make it turn it on with enough disks. Then re-add the remaining.

I have had to do this a number of times when a subset of my sata ports acted up.

Quoted 217 lines · Theophanis Kontogiannis

Dear List.

Hello.

I have the following problem with my CEntOS 6.5 RAID-5

Array of five disks.

At some point in time I added the fifth disk, however I do not remember if I added it as a spare or to go up to RAID-6.

The power and faulty UPS caught me so my RAID-5 has failed (alongside with the on board controller :) )

After replacing the MB, I am in the following situation:

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 21 18:06:31 2014
Raid Level : raid5
Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
Raid Devices : 5
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sat Jul 12 22:11:39 2014
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : tweety.example.com:0 (local to host
tweety.example.com)
UUID : 953836cd:23314476:5db06922:c886893d
Events : 16855

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 32 1 active sync /dev/sdc
2 0 0 2 removed
4 8 64 3 active sync /dev/sde
4 0 0 4 removed

Every effort to add at least one more disk ends up in error:

[root ~]# mdadm /dev/md0 --re-add /dev/sdd
mdadm: --re-add for /dev/sdd to /dev/md0 is not possible

I also made sure that the devices are in the correct physical order.

    mdadm --examine /dev/sd[b-f]


        /dev/sdb:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x2
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953260911 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
        Recovery Offset : 0 sectors
                  State : active
            Device UUID : 7117d844:783abda5:093ee4d9:ba0ac2f0

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : 31569c40 - correct
                 Events : 16845

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 0
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sdc:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 18bf4347:08c262ec:694eba7f:eb8e6b26

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : 55e8716a - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 1
           Array State : .AAAA ('A' == active, '.' == missing)
        /dev/sdd:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 4f77c847:2f567632:66bb5600:9fb2eeba

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : b19f455b - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 2
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sde:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 51b43b6e:7bb8a070:f6375540:28e0b75c

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : f80197e3 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 3
           Array State : .A.AA ('A' == active, '.' == missing)
        /dev/sdf:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 9f7b1355:936e12f2:813c83a0:13854a6b

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : cf0bbec1 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 4
           Array State : .A.AA ('A' == active, '.' == missing)

Any chance to revive this array?

--

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#3 lore raw

Hi Roger,

Thank you for the info.

Results:

[root ~#] mdadm --assemble --force /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
mdadm: Marking array /dev/md0 as 'clean'
mdadm: failed to add /dev/sdd to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdf to /dev/md0: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error [root ~#]

md: sdd does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sde>
md: sdf does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sdc>
md: kicking non-fresh sdb from array!
md: unbind<sdb>
md: export_rdev(sdb)
bio: create slab <bio-1> at 1
md/raid:md0: device sdc operational as raid disk 1 md/raid:md0: device sde operational as raid disk 3 md/raid:md0: allocated 5366kB
md/raid:md0: not enough operational devices (3/5 failed) RAID conf printout:
--- level:5 rd:5 wd:2
disk 1, o:1, dev:sdc
disk 3, o:1, dev:sde
md/raid:md0: failed to run raid set.
md: pers->run() failed ...

[root@tweety ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdc[1] sde[4]

      1953263024 blocks super 1.2

unused devices: <none>

Anything else I could do?

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 4 lines · Roger Heflin

mdadm --stop /dev/mdXX stop
mdadm --assemble --force /dev/mdXX <list all devices good or bad that were previously in the array>

Quoted 227 lines

The assemble with force should make it turn it on with enough disks. Then re-add the remaining.

I have had to do this a number of times when a subset of my sata ports acted up.

Quoted 220 lines · Theophanis Kontogiannis

Dear List.

Hello.

I have the following problem with my CEntOS 6.5 RAID-5

Array of five disks.

At some point in time I added the fifth disk, however I do not remember if I added it as a spare or to go up to RAID-6.

The power and faulty UPS caught me so my RAID-5 has failed (alongside with the on board controller :) )

After replacing the MB, I am in the following situation:

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 21 18:06:31 2014
Raid Level : raid5
Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
Raid Devices : 5
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sat Jul 12 22:11:39 2014
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : tweety.example.com:0 (local to host
tweety.example.com)
UUID : 953836cd:23314476:5db06922:c886893d
Events : 16855

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 32 1 active sync /dev/sdc
2 0 0 2 removed
4 8 64 3 active sync /dev/sde
4 0 0 4 removed

Every effort to add at least one more disk ends up in error:

[root ~]# mdadm /dev/md0 --re-add /dev/sdd
mdadm: --re-add for /dev/sdd to /dev/md0 is not possible

I also made sure that the devices are in the correct physical order.

    mdadm --examine /dev/sd[b-f]


        /dev/sdb:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x2
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953260911 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
        Recovery Offset : 0 sectors
                  State : active
            Device UUID : 7117d844:783abda5:093ee4d9:ba0ac2f0

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : 31569c40 - correct
                 Events : 16845

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 0
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sdc:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 18bf4347:08c262ec:694eba7f:eb8e6b26

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : 55e8716a - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 1
           Array State : .AAAA ('A' == active, '.' == missing)
        /dev/sdd:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 4f77c847:2f567632:66bb5600:9fb2eeba

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : b19f455b - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 2
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sde:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 51b43b6e:7bb8a070:f6375540:28e0b75c

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : f80197e3 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 3
           Array State : .A.AA ('A' == active, '.' == missing)
        /dev/sdf:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 9f7b1355:936e12f2:813c83a0:13854a6b

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : cf0bbec1 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 4
           Array State : .A.AA ('A' == active, '.' == missing)

Any chance to revive this array?

--

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

it shows one of the disks as not fresh (sdb) and it shows sdd and sdf as not being found with a valid superblock at all.

You probably need to do some looking around and see if you can figure out where they are or what is going on.

This may help you find them:
mdadm --examine /dev/<device>
you may need to repeat for any and all unused devices you find on the system and then fix whatever caused the disks to be missing or use the device it is now at.

Quoted 49 lines · Theophanis Kontogiannis

Hi Roger,

Thank you for the info.

Results:

[root ~#] mdadm --assemble --force /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
mdadm: Marking array /dev/md0 as 'clean'
mdadm: failed to add /dev/sdd to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdf to /dev/md0: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error [root ~#]

md: sdd does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sde>
md: sdf does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sdc>
md: kicking non-fresh sdb from array!
md: unbind<sdb>
md: export_rdev(sdb)
bio: create slab <bio-1> at 1
md/raid:md0: device sdc operational as raid disk 1 md/raid:md0: device sde operational as raid disk 3 md/raid:md0: allocated 5366kB
md/raid:md0: not enough operational devices (3/5 failed) RAID conf printout:
--- level:5 rd:5 wd:2
disk 1, o:1, dev:sdc
disk 3, o:1, dev:sde
md/raid:md0: failed to run raid set.
md: pers->run() failed ...

[root@tweety ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdc[1] sde[4]

      1953263024 blocks super 1.2

unused devices: <none>

Anything else I could do?

Quoted 239 lines

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 231 lines · Roger Heflin

mdadm --stop /dev/mdXX stop
mdadm --assemble --force /dev/mdXX <list all devices good or bad that were previously in the array>

The assemble with force should make it turn it on with enough disks. Then re-add the remaining.

I have had to do this a number of times when a subset of my sata ports acted up.

Quoted 220 lines · Theophanis Kontogiannis

Dear List.

Hello.

I have the following problem with my CEntOS 6.5 RAID-5

Array of five disks.

At some point in time I added the fifth disk, however I do not remember if I added it as a spare or to go up to RAID-6.

The power and faulty UPS caught me so my RAID-5 has failed (alongside with the on board controller :) )

After replacing the MB, I am in the following situation:

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 21 18:06:31 2014
Raid Level : raid5
Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
Raid Devices : 5
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sat Jul 12 22:11:39 2014
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : tweety.example.com:0 (local to host
tweety.example.com)
UUID : 953836cd:23314476:5db06922:c886893d
Events : 16855

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 32 1 active sync /dev/sdc
2 0 0 2 removed
4 8 64 3 active sync /dev/sde
4 0 0 4 removed

Every effort to add at least one more disk ends up in error:

[root ~]# mdadm /dev/md0 --re-add /dev/sdd
mdadm: --re-add for /dev/sdd to /dev/md0 is not possible

I also made sure that the devices are in the correct physical order.

    mdadm --examine /dev/sd[b-f]


        /dev/sdb:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x2
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953260911 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
        Recovery Offset : 0 sectors
                  State : active
            Device UUID : 7117d844:783abda5:093ee4d9:ba0ac2f0

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : 31569c40 - correct
                 Events : 16845

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 0
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sdc:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 18bf4347:08c262ec:694eba7f:eb8e6b26

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : 55e8716a - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 1
           Array State : .AAAA ('A' == active, '.' == missing)
        /dev/sdd:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 4f77c847:2f567632:66bb5600:9fb2eeba

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : b19f455b - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 2
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sde:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 51b43b6e:7bb8a070:f6375540:28e0b75c

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : f80197e3 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 3
           Array State : .A.AA ('A' == active, '.' == missing)
        /dev/sdf:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 9f7b1355:936e12f2:813c83a0:13854a6b

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : cf0bbec1 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 4
           Array State : .A.AA ('A' == active, '.' == missing)

Any chance to revive this array?

--

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#5 lore raw

Hi Roger,

Hi All,

Apparently this is the reason I can not solve it out myself.

According to the following all looks OK and all devices are in place, clean, with correct SB version and in the correct physical sequence (correct me if I am wrong):

mdadm --examine /dev/sd[b-f]|grep Version

        Version : 1.2
        Version : 1.2
        Version : 1.2
        Version : 1.2
        Version : 1.2

mdadm --examine /dev/sd[b-f]|grep Role
Device Role : Active device 0
Device Role : Active device 1
Device Role : Active device 2
Device Role : Active device 3
Device Role : Active device 4

mdadm --examine /dev/sd[b-f]|grep Event

         Events : 16845
         Events : 16855
         Events : 16855
         Events : 16855
         Events : 16855

mdadm --examine /dev/sd[b-f]|grep Update

    Update Time : Sat Jul 12 21:32:42 2014
    Update Time : Sat Jul 12 22:11:39 2014
    Update Time : Sat Jul 12 21:32:42 2014
    Update Time : Sat Jul 12 22:11:39 2014
    Update Time : Sat Jul 12 22:11:39 2014

However this is what confuses me and maybe is the reason of my problem. However I can not evaluate it:

mdadm --examine /dev/sd[b-f]|grep State
State : active
Array State : AAAAA ('A' == active, '.' == missing)
State : clean
Array State : .AAAA ('A' == active, '.' == missing)
State : active
Array State : AAAAA ('A' == active, '.' == missing)
State : active
Array State : .A.AA ('A' == active, '.' == missing)
State : clean
Array State : .A.AA ('A' == active, '.' == missing)

Any ideas?

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 6 lines · Roger Heflin

it shows one of the disks as not fresh (sdb) and it shows sdd and sdf as not being found with a valid superblock at all.

You probably need to do some looking around and see if you can figure out where they are or what is going on.

Quoted 295 lines

This may help you find them:
mdadm --examine /dev/<device>
you may need to repeat for any and all unused devices you find on the system and then fix whatever caused the disks to be missing or use the device it is now at.

Quoted 287 lines · Theophanis Kontogiannis

Hi Roger,

Thank you for the info.

Results:

[root ~#] mdadm --assemble --force /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
mdadm: Marking array /dev/md0 as 'clean'
mdadm: failed to add /dev/sdd to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdf to /dev/md0: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error [root ~#]

md: sdd does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sde>
md: sdf does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sdc>
md: kicking non-fresh sdb from array!
md: unbind<sdb>
md: export_rdev(sdb)
bio: create slab <bio-1> at 1
md/raid:md0: device sdc operational as raid disk 1 md/raid:md0: device sde operational as raid disk 3 md/raid:md0: allocated 5366kB
md/raid:md0: not enough operational devices (3/5 failed) RAID conf printout:
--- level:5 rd:5 wd:2
disk 1, o:1, dev:sdc
disk 3, o:1, dev:sde
md/raid:md0: failed to run raid set.
md: pers->run() failed ...

[root@tweety ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdc[1] sde[4]

      1953263024 blocks super 1.2

unused devices: <none>

Anything else I could do?

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 231 lines · Roger Heflin

mdadm --stop /dev/mdXX stop
mdadm --assemble --force /dev/mdXX <list all devices good or bad that were previously in the array>

The assemble with force should make it turn it on with enough disks. Then re-add the remaining.

I have had to do this a number of times when a subset of my sata ports acted up.

Quoted 220 lines · Theophanis Kontogiannis

Dear List.

Hello.

I have the following problem with my CEntOS 6.5 RAID-5

Array of five disks.

At some point in time I added the fifth disk, however I do not remember if I added it as a spare or to go up to RAID-6.

The power and faulty UPS caught me so my RAID-5 has failed (alongside with the on board controller :) )

After replacing the MB, I am in the following situation:

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 21 18:06:31 2014
Raid Level : raid5
Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
Raid Devices : 5
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sat Jul 12 22:11:39 2014
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : tweety.example.com:0 (local to host
tweety.example.com)
UUID : 953836cd:23314476:5db06922:c886893d
Events : 16855

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 32 1 active sync /dev/sdc
2 0 0 2 removed
4 8 64 3 active sync /dev/sde
4 0 0 4 removed

Every effort to add at least one more disk ends up in error:

[root ~]# mdadm /dev/md0 --re-add /dev/sdd
mdadm: --re-add for /dev/sdd to /dev/md0 is not possible

I also made sure that the devices are in the correct physical order.

    mdadm --examine /dev/sd[b-f]


        /dev/sdb:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x2
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953260911 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
        Recovery Offset : 0 sectors
                  State : active
            Device UUID : 7117d844:783abda5:093ee4d9:ba0ac2f0

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : 31569c40 - correct
                 Events : 16845

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 0
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sdc:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 18bf4347:08c262ec:694eba7f:eb8e6b26

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : 55e8716a - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 1
           Array State : .AAAA ('A' == active, '.' == missing)
        /dev/sdd:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 4f77c847:2f567632:66bb5600:9fb2eeba

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : b19f455b - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 2
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sde:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 51b43b6e:7bb8a070:f6375540:28e0b75c

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : f80197e3 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 3
           Array State : .A.AA ('A' == active, '.' == missing)
        /dev/sdf:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 9f7b1355:936e12f2:813c83a0:13854a6b

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : cf0bbec1 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 4
           Array State : .A.AA ('A' == active, '.' == missing)

Any chance to revive this array?

--

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#6 lore raw
In reply to Roger Heflin

Hi Roger, Hi List,

With a high heart bit rate I tried this:

mdadm --create /dev/md0 --assume-clean --level=5 --raid-devices=5 /dev/sdb /dev/sdc /dev/sde /dev/sdd missing

IT WORKED!

Now I have mounted my date back.

BUT:

1. I have one failing disk according to SMART (that is /dev/sdc)

2. /dev/sdb and /dev/sdd have a little smaller size then the others:

blockdev --getsz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf 1953523055
1953525168
1953523055
1953525168
1953523055

The idea to re-create the array came from here:

http://ubuntuforums.org/showthread.php?t=1736742&p=10708294#post10708294

Now I will replace the two different disks and will upgrade to RAID-6.

Any hints what might have been the issue?

As mentioned on the exactly previous post, all seamed to be in order with the array.

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 6 lines · Roger Heflin

it shows one of the disks as not fresh (sdb) and it shows sdd and sdf as not being found with a valid superblock at all.

You probably need to do some looking around and see if you can figure out where they are or what is going on.

Quoted 295 lines

This may help you find them:
mdadm --examine /dev/<device>
you may need to repeat for any and all unused devices you find on the system and then fix whatever caused the disks to be missing or use the device it is now at.

Quoted 287 lines · Theophanis Kontogiannis

Hi Roger,

Thank you for the info.

Results:

[root ~#] mdadm --assemble --force /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
mdadm: Marking array /dev/md0 as 'clean'
mdadm: failed to add /dev/sdd to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdf to /dev/md0: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error [root ~#]

md: sdd does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sde>
md: sdf does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sdc>
md: kicking non-fresh sdb from array!
md: unbind<sdb>
md: export_rdev(sdb)
bio: create slab <bio-1> at 1
md/raid:md0: device sdc operational as raid disk 1 md/raid:md0: device sde operational as raid disk 3 md/raid:md0: allocated 5366kB
md/raid:md0: not enough operational devices (3/5 failed) RAID conf printout:
--- level:5 rd:5 wd:2
disk 1, o:1, dev:sdc
disk 3, o:1, dev:sde
md/raid:md0: failed to run raid set.
md: pers->run() failed ...

[root@tweety ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdc[1] sde[4]

      1953263024 blocks super 1.2

unused devices: <none>

Anything else I could do?

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 231 lines · Roger Heflin

mdadm --stop /dev/mdXX stop
mdadm --assemble --force /dev/mdXX <list all devices good or bad that were previously in the array>

The assemble with force should make it turn it on with enough disks. Then re-add the remaining.

I have had to do this a number of times when a subset of my sata ports acted up.

Quoted 220 lines · Theophanis Kontogiannis

Dear List.

Hello.

I have the following problem with my CEntOS 6.5 RAID-5

Array of five disks.

At some point in time I added the fifth disk, however I do not remember if I added it as a spare or to go up to RAID-6.

The power and faulty UPS caught me so my RAID-5 has failed (alongside with the on board controller :) )

After replacing the MB, I am in the following situation:

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 21 18:06:31 2014
Raid Level : raid5
Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
Raid Devices : 5
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sat Jul 12 22:11:39 2014
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : tweety.example.com:0 (local to host
tweety.example.com)
UUID : 953836cd:23314476:5db06922:c886893d
Events : 16855

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 32 1 active sync /dev/sdc
2 0 0 2 removed
4 8 64 3 active sync /dev/sde
4 0 0 4 removed

Every effort to add at least one more disk ends up in error:

[root ~]# mdadm /dev/md0 --re-add /dev/sdd
mdadm: --re-add for /dev/sdd to /dev/md0 is not possible

I also made sure that the devices are in the correct physical order.

    mdadm --examine /dev/sd[b-f]


        /dev/sdb:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x2
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953260911 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
        Recovery Offset : 0 sectors
                  State : active
            Device UUID : 7117d844:783abda5:093ee4d9:ba0ac2f0

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : 31569c40 - correct
                 Events : 16845

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 0
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sdc:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 18bf4347:08c262ec:694eba7f:eb8e6b26

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : 55e8716a - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 1
           Array State : .AAAA ('A' == active, '.' == missing)
        /dev/sdd:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 4f77c847:2f567632:66bb5600:9fb2eeba

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : b19f455b - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 2
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sde:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 51b43b6e:7bb8a070:f6375540:28e0b75c

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : f80197e3 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 3
           Array State : .A.AA ('A' == active, '.' == missing)
        /dev/sdf:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 9f7b1355:936e12f2:813c83a0:13854a6b

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : cf0bbec1 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 4
           Array State : .A.AA ('A' == active, '.' == missing)

Any chance to revive this array?

--

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw

I try to avoid the create as if wrong (order wise) things get risky. So far even when I have lost a number of disks the --force has worked for me (controller failure dropped 3 disks).

One thing in the future would be to partition all of the devices, the 2 devices being a slightly different size may be what was causing it
issues. With partitioning you should be able to setup all of the
devices to appear to be the exact same size.

Quoted 35 lines · Theophanis Kontogiannis

Hi Roger, Hi List,

With a high heart bit rate I tried this:

mdadm --create /dev/md0 --assume-clean --level=5 --raid-devices=5 /dev/sdb /dev/sdc /dev/sde /dev/sdd missing

IT WORKED!

Now I have mounted my date back.

BUT:

1. I have one failing disk according to SMART (that is /dev/sdc)

2. /dev/sdb and /dev/sdd have a little smaller size then the others:

blockdev --getsz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf 1953523055
1953525168
1953523055
1953525168
1953523055

The idea to re-create the array came from here:

http://ubuntuforums.org/showthread.php?t=1736742&p=10708294#post10708294

Now I will replace the two different disks and will upgrade to RAID-6.

Any hints what might have been the issue?

As mentioned on the exactly previous post, all seamed to be in order with the array.

Quoted 309 lines

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 301 lines · Roger Heflin

it shows one of the disks as not fresh (sdb) and it shows sdd and sdf as not being found with a valid superblock at all.

You probably need to do some looking around and see if you can figure out where they are or what is going on.

This may help you find them:
mdadm --examine /dev/<device>
you may need to repeat for any and all unused devices you find on the system and then fix whatever caused the disks to be missing or use the device it is now at.

Quoted 287 lines · Theophanis Kontogiannis

Hi Roger,

Thank you for the info.

Results:

[root ~#] mdadm --assemble --force /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
mdadm: Marking array /dev/md0 as 'clean'
mdadm: failed to add /dev/sdd to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdf to /dev/md0: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error [root ~#]

md: sdd does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sde>
md: sdf does not have a valid v1.2 superblock, not importing!
md: md_import_device returned -22
md: bind<sdc>
md: kicking non-fresh sdb from array!
md: unbind<sdb>
md: export_rdev(sdb)
bio: create slab <bio-1> at 1
md/raid:md0: device sdc operational as raid disk 1 md/raid:md0: device sde operational as raid disk 3 md/raid:md0: allocated 5366kB
md/raid:md0: not enough operational devices (3/5 failed) RAID conf printout:
--- level:5 rd:5 wd:2
disk 1, o:1, dev:sdc
disk 3, o:1, dev:sde
md/raid:md0: failed to run raid set.
md: pers->run() failed ...

[root@tweety ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdc[1] sde[4]

      1953263024 blocks super 1.2

unused devices: <none>

Anything else I could do?

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

Quoted 231 lines · Roger Heflin

mdadm --stop /dev/mdXX stop
mdadm --assemble --force /dev/mdXX <list all devices good or bad that were previously in the array>

The assemble with force should make it turn it on with enough disks. Then re-add the remaining.

I have had to do this a number of times when a subset of my sata ports acted up.

Quoted 220 lines · Theophanis Kontogiannis

Dear List.

Hello.

I have the following problem with my CEntOS 6.5 RAID-5

Array of five disks.

At some point in time I added the fifth disk, however I do not remember if I added it as a spare or to go up to RAID-6.

The power and faulty UPS caught me so my RAID-5 has failed (alongside with the on board controller :) )

After replacing the MB, I am in the following situation:

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 21 18:06:31 2014
Raid Level : raid5
Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
Raid Devices : 5
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sat Jul 12 22:11:39 2014
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : tweety.example.com:0 (local to host
tweety.example.com)
UUID : 953836cd:23314476:5db06922:c886893d
Events : 16855

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 32 1 active sync /dev/sdc
2 0 0 2 removed
4 8 64 3 active sync /dev/sde
4 0 0 4 removed

Every effort to add at least one more disk ends up in error:

[root ~]# mdadm /dev/md0 --re-add /dev/sdd
mdadm: --re-add for /dev/sdd to /dev/md0 is not possible

I also made sure that the devices are in the correct physical order.

    mdadm --examine /dev/sd[b-f]


        /dev/sdb:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x2
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953260911 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
        Recovery Offset : 0 sectors
                  State : active
            Device UUID : 7117d844:783abda5:093ee4d9:ba0ac2f0

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : 31569c40 - correct
                 Events : 16845

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 0
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sdc:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 18bf4347:08c262ec:694eba7f:eb8e6b26

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : 55e8716a - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 1
           Array State : .AAAA ('A' == active, '.' == missing)
        /dev/sdd:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 4f77c847:2f567632:66bb5600:9fb2eeba

            Update Time : Sat Jul 12 21:32:42 2014
               Checksum : b19f455b - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 2
           Array State : AAAAA ('A' == active, '.' == missing)
        /dev/sde:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : active
            Device UUID : 51b43b6e:7bb8a070:f6375540:28e0b75c

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : f80197e3 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 3
           Array State : .A.AA ('A' == active, '.' == missing)
        /dev/sdf:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x0
             Array UUID : 953836cd:23314476:5db06922:c886893d
                   Name : tweety.example.com:0  (local to host
        tweety.example.com)
          Creation Time : Fri Feb 21 18:06:31 2014
             Raid Level : raid5
           Raid Devices : 5

         Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
             Array Size : 3906521088 (3725.55 GiB 4000.28 GB)
          Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 9f7b1355:936e12f2:813c83a0:13854a6b

            Update Time : Sat Jul 12 22:11:39 2014
               Checksum : cf0bbec1 - correct
                 Events : 16855

                 Layout : left-symmetric
             Chunk Size : 512K

           Device Role : Active device 4
           Array State : .A.AA ('A' == active, '.' == missing)

Any chance to revive this array?

--

Kind Regards,
Theophanis Kontogiannis

ΜΦΧ
ΘΚ

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw
In reply to Theophanis Kontogiannis
Quoted 14 lines · Theophanis Kontogiannis

Hi Roger, Hi List,

With a high heart bit rate I tried this:

mdadm --create /dev/md0 --assume-clean --level=5 --raid-devices=5 /dev/sdb /dev/sdc /dev/sde /dev/sdd missing

IT WORKED!

Now I have mounted my date back.

BUT:

1. I have one failing disk according to SMART (that is /dev/sdc)

Quoted 9 lines

2. /dev/sdb and /dev/sdd have a little smaller size then the others:

blockdev --getsz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf 1953523055
1953525168
1953523055
1953525168
1953523055

All 1TB disks should be exactly the same size (manufacturers are using standard sizes for all disks nowadays). Are these drives all on the same controllers? This looks like maybe the BIOS is reserving some space on the disks, which may have overwritten something in the original array.

Have you run a fsck across the array after reassembling it? I suspect you may find the filesystem size doesn't match the original one.

Regards,
Robin

Signature · 4 lines
     ___        
    ( ' }     |       Robin Hill        <[email protected]> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |