assistance recovering failed raid6 array
I am running a software raid6 with 36 x 3TB disks (sda to sdaj). All disks have one partition (gpt, 100%, primary, raid on) and i am using btrfs on top of the raid.
Last week one of the disks failed and was unrecoverable. I replaced the disk (sdk) with a new one and the resync process started.
At around 80% recovery two further disks failed and the recovery process was stopped. That failed disks are sdm and sdh.
All other disks seem to be fine and I was about the use the "mdadm --create" command when i remembered the lines
"You have been warned! It's better to send an email to the linux-raid mailing list with detailed information"
So here i am for an advice how to continue.
More details:
Only 35% of the raid space is used.
The disks status is:
sdk: original disk is dead and the replacement was around 80% recovered.
sdm: i was able to copy the first 2 TB with two errors (128kbyte) and the third TB with around 200GB missing data using ddrescue to a new disk.
sdh: the original disk is dead and i replaced it with a brand new one and created the partition sdh1.
Since the array is offline i cannot add sdh1 to the raid and trying to assemble the array gives me:
For mdadm --assemble --force with sdh1:
mdadm: no RAID superblock on /dev/sdh1
mdadm: /dev/sdh1 has no superblock - assembly aborted
For mdadm --assemble --force without sdh1:
mdadm: /dev/md0 assembled from 33 drives, 1 rebuilding and 1 spare - not enough to start the array.
Full status of /dev/sda1:
mdadm --examine /dev/sda1/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : f90e9c41:5aa3c3b2:d715781b:1abbb439
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b0b57ef2 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active,
'.' == missing, 'R' == replacing)
mdadm --examine for each drive to get "Device Role":
"sda Device Role : Active device 0" "sdb Device Role : Active device 1" "sdc Device Role : Active device 2" "sdd Device Role : Active device 3" "sde Device Role : Active device 4" "sdf Device Role : Active device 5" "sdg Device Role : Active device 6" "sdh" mdadm: No md superblock detected on /dev/sdh1. "sdi Device Role : Active device 8" "sdj Device Role : Active device 9" "sdk Device Role : spare" "sdl Device Role : Active device 11" "sdm Device Role : Active device 12" "sdn Device Role : Active device 13" "sdo Device Role : Active device 14" "sdp Device Role : Active device 15" "sdq Device Role : Active device 16" "sdr Device Role : Active device 17" "sds Device Role : Active device 18" "sdt Device Role : Active device 19" "sdu Device Role : Active device 20" "sdv Device Role : Active device 21" "sdw Device Role : Active device 22" "sdx Device Role : Active device 23" "sdy Device Role : Active device 24" "sdz Device Role : Active device 25" "sdaa Device Role : Active device 26" "sdab Device Role : Active device 27" "sdac Device Role : Active device 28" "sdad Device Role : Active device 29" "sdae Device Role : Active device 30" "sdaf Device Role : Active device 31" "sdag Device Role : Active device 32" "sdah Device Role : Active device 33" "sdai Device Role : Active device 34" "sdaj Device Role : Active device 35"
The system is Ubuntu 16.04.2 LTS (x86_64) with a 4.4.0-62-generic kernel.
mdadm --version gives me: mdadm - v3.3 - 3rd September 2013
Signature · 1 line
<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>
Quoted 9 lines · Martin Bosner
I am running a software raid6 with 36 x 3TB disks (sda to sdaj). All disks have one partition (gpt, 100%, primary, raid on) and i am using btrfs on top of the raid.
Last week one of the disks failed and was unrecoverable. I replaced the disk (sdk) with a new one and the resync process started.
At around 80% recovery two further disks failed and the recovery process was stopped. That failed disks are sdm and sdh.
Quoted 6 lines
All other disks seem to be fine and I was about the use the "mdadm --create" command when i remembered the lines
"You have been warned! It's better to send an email to the linux-raid mailing list with detailed information"So here i am for an advice how to continue.
More information, please. Paste inline, untrimmed, in your reply with line wrapping disabled. Plain text only. Use multiple mails if needed. List limit is ~100k IIRC.
# dmesg
# for x in /dev/sd[a-z] /dev/sda[a-j] ; do echo mdadm -E ${x}1 ; smartctl -iA -l scterc $x ; donePhil
Hi Martin,
{ Note: convention on kernel.org is reply-to-all, bottom post or interleave, and trim unneeded material. }
Quoted 1 line · Martin Bosner
for x in /dev/sd[a-z] /dev/sda[a-j] ; do echo mdadm -E ${x}1 ; smartctl -iA -l scterc $x ; done
Darn. I didn't mean to leave 'echo' there. Please run this part over again:
for x in /dev/sd[a-z] /dev/sda[a-j] ; do mdadm -E ${x}1 ; done
Quoted 49 lines
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-62-generic] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-1CH166 Serial Number: Z1F4RN82 LU WWN Device Id: 5 000c50 0662080a6 Firmware Version: CC27 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Feb 20 17:04:09 2017 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled=== START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 111 099 006 Pre-fail Always - 32613624 3 Spin_Up_Time 0x0003 098 098 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 4 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 067 060 030 Pre-fail Always - 5379264 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 4222 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 4 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1 190 Airflow_Temperature_Cel 0x0022 072 066 045 Old_age Always - 28 (Min/Max 26/28) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 4 194 Temperature_Celsius 0x0022 028 040 000 Old_age Always - 28 (0 25 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 4222h+01m+40.011s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 6286190299 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 103173878942
Quoted 1 line
SCT Error Recovery Control command not supported
^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^
Eewwww! You have desktop drives. Which means your array has blown up due to timeout mismatch. You have some reading to do.[1]
Phil
[1] Recommendations from the archives (whole threads):
http://marc.info/?l=linux-raid&m=139050322510249&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=132477199207506
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142487508806844&w=3
http://marc.info/?l=linux-raid&m=144535576302583&w=2
Hi Phil,
Quoted 2 lines
{ Note: convention on kernel.org is reply-to-all, bottom post or interleave, and trim unneeded material. }
Sorry, hope this one is better.
Quoted 1 line
for x in /dev/sd[a-z] /dev/sda[a-j] ; do mdadm -E ${x}1 ; done
See below for full output.
Quoted 3 lines
Eewwww! You have desktop drives. Which means your array has blown up due to timeout mismatch. You have some reading to do.[1]
I will be using so called “NAS” or “enterprise disks in the next cluster … especially these seagate disks were a bad decision.
Quoted 2 lines
http://marc.info/?l=linux-raid&m=142487508806844&w=3
http://marc.info/?l=linux-raid&m=144535576302583&w=2
I will read through it.
Martin
#########
for x in /dev/sd[a-z] /dev/sda[a-j] ; do mdadm -E ${x}1 ; done /dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : f90e9c41:5aa3c3b2:d715781b:1abbb439
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b0b57ef2 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : bf3081d1:5afead3a:839df956:098403b9
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 83554c55 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 6e592519:9dafe1fd:6616a6b4:6de9ab52
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 8582048f - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 465ca8a2:45bd05a3:94b4d0cc:0bf9ee5d
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d796c2dc - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 79f7fb50:15f0887f:4d7adf0c:632e9743
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 88248bf0 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 5a1097b1:7da38314:8633c3f9:d3b843ab
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d24a9be4 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 31076c1e:97f2be2f:1ae76487:48667fc7
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 43842df - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/sdh1./dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 34ecbf22:16e15f2f:a9075b03:6f8f2de3
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9fd16018 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 390cbe6a:bceed865:5d88091f:86c7228b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e1ec468f - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 79b27fa7:954302fe:4f669a20:1ddf9a15
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 42e0375d - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : c7215fad:1c6ecbbf:2c2e0feb:aabbb208
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : c8157597 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x8b
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Recovery Offset : 296160 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 1366f174:d65cf0f7:b20f5b4f:c2263bbe
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : e1a5d782 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : c9b9b4ad:74b4989c:1c0003cd:dd402919
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 97a2aaf2 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 13
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : d5a5ec07:91695b21:087ebb12:f3e6bf3f
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e2ec701c - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 14
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdp1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 7f4f1b78:7a0ae004:664b2208:da13eed6
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : c334b517 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 15
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdq1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : a2158ac6:82ac5c70:31d9b49b:66077e36
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 70429e7a - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 16
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdr1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : af23e4bb:04bae1e3:6b79aa24:ebf1ccb3
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : df6644c9 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 17
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sds1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 0c1675d5:0e2e9b88:3ab7545c:f5f2cd99
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : bb5bea0a - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 18
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdt1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : f2920d03:7784140f:0108652e:ef335243
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : eb024f19 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 19
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdu1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 00b49bc7:54fa2eb2:38be88f4:177812ad
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 828ee067 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 20
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdv1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : a04149c8:d99b98d4:6ad76d27:b6930004
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2f79445d - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 21
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdw1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 3609155b:7272da25:a6ccf32b:707e27e1
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : f533c282 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 22
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdx1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : c5462d71:6f69ab33:a6ddeb6a:28210c6e
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : e4f9aae3 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 23
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdy1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 44823783:34d0edf7:03d4ae6f:522befc6
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 18ec4da1 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 24
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdz1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 063185d5:e0d3dbce:ab39c4fa:ae855b0c
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 12a9c008 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 25
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdaa1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : da6a846c:563d61bf:ee1e786a:c9e5280b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 8afa8c7 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 26
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdab1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 0ea3c624:8e2e59ab:ed195520:23917089
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : e10e7875 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 27
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdac1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : e6c66c40:e0c49813:e87dc481:7b57ef8b
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c8e25cf3 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 28
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdad1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : af24f1fb:a55ddb8b:3ecd2971:6f9dc92d
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 8de8e8dd - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 29
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdae1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : d4e87abe:00003a9d:e950d72c:f5b93939
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 28eeef8f - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 30
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdaf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 2ea54072:67978e8b:c6b99895:76f4157f
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 79a6e6b2 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 31
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdag1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 52ee4e92:9a60fdd5:b3393a70:f0e8e53f
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 7f956d5e - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 32
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdah1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 80818d5a:85638be1:f9974e08:081404d2
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors - bad blocks present.
Checksum : 7d948cde - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 33
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdai1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 6f014106:2eea458c:4a6dcf1a:a2666db0
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c4ecbb59 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 34
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)/dev/sdaj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Name : media-storage:0 (local to host media-storage) Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Raid Devices : 36
Avail Dev Size : 5860268032 (2794.39 GiB 3000.46 GB)
Array Size : 99624556544 (95009.38 GiB 102015.55 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : 0fdc0ec2:42d30dcb:f00d2c63:26171824
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Feb 15 14:08:28 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 7b89d039 - correct
Events : 140559
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 35
Array State : AAAAAAA.AA.AAAAAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
Quoted 4 lines · Martin Bosner
Quoted 3 lines
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-1CH166
So you have the most terrible hard drive possible [1][2](they WILL ALL fail), ran in about the most terrible RAID setup possible (only a single RAID5 would have been worse). Now you realize why the latter was a bad idea: with such a great number of disks, should have picked a 3x12-member RAID6 or similar. Just let this be a lesson in component choice and risk assessment, restore from your backups and move on.
[1] https://en.wikipedia.org/wiki/ST3000DM001
[2] https://www.quora.com/What-are-some-of-the-worst-hard-drive-designs-ever
Signature · 2 lines
With respect, Roman
Hi Martin,
Quoted 10 lines · Martin Bosner
Hi Phil,
Quoted 2 lines
{ Note: convention on kernel.org is reply-to-all, bottom post or interleave, and trim unneeded material. }
Sorry, hope this one is better.
Quoted 1 line
for x in /dev/sd[a-z] /dev/sda[a-j] ; do mdadm -E ${x}1 ; done
See below for full output.
Not good.
Quoted 2 lines
Quoted 2 lines
Eewwww! You have desktop drives. Which means your array has blown up due to timeout mismatch. You have some reading to do.[1]
Of the 36 original disks, you have 34. You have one incomplete rebuild, meaning it is still technically a spare. One of the still active 34 is also showing pending relocations, meaning that disk will not be able to supply all sectors to complete any recovery. { /dev/sdah, serial # S1F0FPYR }
If you have any access to the two "dead" drives, there might be a slight chance. Since they were likely kicked out due to timeout mismatch, not a complete failure, this could be possible.
Otherwise, you are utterly screwed. Sorry.
Quoted 2 lines
I will be using so called “NAS” or “enterprise disks in the next cluster … especially these seagate disks were a bad decision.
Yes.
Phil
Quoted 1 line
So you have the most terrible hard drive possible [1][2](they WILL ALL fail),
That is very true.
Quoted 9 lines · Phil Turmel
Of the 36 original disks, you have 34. You have one incomplete rebuild, meaning it is still technically a spare. One of the still active 34 is also showing pending relocations, meaning that disk will not be able to supply all sectors to complete any recovery. {
/dev/sdah, serial # S1F0FPYR }
Quoted 5 lines
If you have any access to the two "dead" drives, there might be a slight chance. Since they were likely kicked out due to timeout mismatch, not a complete failure, this could be possible.
Otherwise, you are utterly screwed. Sorry.
The disks are dead. I already tried different boards but that did not help.
What would happen if i recreate the array with —assume-clean ? Would i be able to start the array? Can I mark disks as clean? I actually have one failed disks, one nearly recovered disk and one that has been copied by 2/3 ...
Martin
Quoted 9 lines · Martin Bosner
Quoted 9 lines · Phil Turmel
Of the 36 original disks, you have 34. You have one incomplete rebuild, meaning it is still technically a spare. One of the still active 34 is also showing pending relocations, meaning that disk will not be able to supply all sectors to complete any recovery. {
/dev/sdah, serial # S1F0FPYR }
Quoted 7 lines
Quoted 5 lines
If you have any access to the two "dead" drives, there might be a slight chance. Since they were likely kicked out due to timeout mismatch, not a complete failure, this could be possible.
Otherwise, you are utterly screwed. Sorry.
The disks are dead. I already tried different boards but that did not help.
If it had been the timeout problem, you would probably have been able to recover the array. As it isn't :-(
Quoted 3 lines
What would happen if i recreate the array with —assume-clean ? Would i be able to start the array? Can I mark disks as clean? I actually have one failed disks, one nearly recovered disk and one that has been copied by 2/3 ...
You can try "--assemble --force". It sounds like you might well get away with it.
BUT! DO NOT ATTEMPT TO USE THE ARRAY IF IT LOOKS LIKE IT'S OKAY. Are all the disks the same age? In which case, all the other drives are on the verge of failure, too!
I don't know whether to suggest you use smartctl to see what state the drives are in (I've seen too many reports of allegedly healthy drives failing, so I wouldn't trust it, especially with this particular drive.)
ddrescue your remaining drives *now*, and hope you're okay. You say you have 35GB, across 36 drives, so you should only be using the first 1GB of each drive. We hope ...
Quoted 2 lines
Martin
Cheers,
Wol
Quoted 5 lines · Wols Lists
Quoted 1 line
You can try "--assemble --force". It sounds like you might well get away with it.
Would it be possible to start the array by adding sdk1 (setting state as active) and resetting the state of sdm1? The array failed while i was copying stuff to another place ...
With —assemble —force i get this:
mdadm --detail /dev/md0/dev/md0:
Version : 1.2
Creation Time : Sun Sep 18 22:46:42 2016
Raid Level : raid6
Used Dev Size : -1
Raid Devices : 36
Total Devices : 35
Persistence : Superblock is persistent
Update Time : Wed Feb 15 14:08:28 2017
State : active, FAILED, Not Started
Active Devices : 33
Working Devices : 35
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 512K
Name : media-storage:0 (local to host media-storage) UUID : 5c7c227e:22de5fc1:ca3ebb65:9c283567
Events : 140559
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1
6 8 97 6 active sync /dev/sdg1
14 0 0 14 removed
8 8 129 8 active sync /dev/sdi1
9 8 145 9 active sync /dev/sdj1
20 0 0 20 removed
39 8 177 11 active sync /dev/sdl1
12 8 193 12 spare rebuilding /dev/sdm1
13 8 209 13 active sync /dev/sdn1
14 8 225 14 active sync /dev/sdo1
40 8 241 15 active sync /dev/sdp1
16 65 1 16 active sync /dev/sdq1
17 65 17 17 active sync /dev/sdr1
18 65 33 18 active sync /dev/sds1
19 65 49 19 active sync /dev/sdt1
20 65 65 20 active sync /dev/sdu1
21 65 81 21 active sync /dev/sdv1
22 65 97 22 active sync /dev/sdw1
43 65 113 23 active sync /dev/sdx1
36 65 129 24 active sync /dev/sdy1
25 65 145 25 active sync /dev/sdz1
41 65 161 26 active sync /dev/sdaa1
27 65 177 27 active sync /dev/sdab1
28 65 193 28 active sync /dev/sdac1
37 65 209 29 active sync /dev/sdad1
38 65 225 30 active sync /dev/sdae1
42 65 241 31 active sync /dev/sdaf1
32 66 1 32 active sync /dev/sdag1
33 66 17 33 active sync /dev/sdah1
34 66 33 34 active sync /dev/sdai1
35 66 49 35 active sync /dev/sdaj1
44 8 161 - spare /dev/sdk1
Cheers
Martin
Quoted 8 lines · Martin Bosner
Quoted 5 lines
If you have any access to the two "dead" drives, there might be a slight chance. Since they were likely kicked out due to timeout mismatch, not a complete failure, this could be possible.
Otherwise, you are utterly screwed. Sorry.
The disks are dead. I already tried different boards but that did not help.
Oh, well.
Quoted 4 lines
What would happen if i recreate the array with —assume-clean ? Would i be able to start the array? Can I mark disks as clean? I actually have one failed disks, one nearly recovered disk and one that has been copied by 2/3 ...
For every stripe in the array, you need 34 devices of 36 to be readable. Any time you fall back on ddrescue to make one of those 34, you are ensuring that some data is lost. But yes, that would otherwise work. The 2/3 recovered disk is only useful in this (use ddrescue to get as much of the missing disk as possible).
If you can get to 35 of 36 original disks, even with scattered errors, you could complete a check scrub to make 35 good disks. With timeout mismatch, you'll have to override the kernel timeouts for all devices, so such a scrub would take a very long time, but would recover everything.
Quoted 1 line
Martin
Phil
Quoted 7 lines · Phil Turmel
If you can get to 35 of 36 original disks, even with scattered errors, you could complete a check scrub to make 35 good disks. With timeout mismatch, you'll have to override the kernel timeouts for all devices, so such a scrub would take a very long time, but would recover everything.
Is there a way tell mdadm to use sdk1 as a “active” device? And how can i tell the array that it should not try to recover sdm1 but set it active ? Is there any magic to force the state ? It might not be healthy for the normal use case but might be helpful for me.
Martin
Quoted 5 lines · Phil Turmel
For every stripe in the array, you need 34 devices of 36 to be readable. Any time you fall back on ddrescue to make one of those 34, you are ensuring that some data is lost. But yes, that would otherwise work. The 2/3 recovered disk is only useful in this (use ddrescue to get as much of the missing disk as possible).
I keep on asking :-)
But there's a request on the linux wiki program for someone to write a utility program that takes a ddrescue log and flags the duff sectors as "soft unreadable". That would mean that if you can recover 35 drives, provided no stripe has lost two sectors across two drives, you wouldn't lose any data.
If you want to try and write that utility? Or if you want to email me a ddrescue log with a bunch of failed sectors, I'll have a go at writing it myself :-)
Cheers,
Wol
Quoted 9 lines · Wols Lists
But there's a request on the linux wiki program for someone to write a utility program that takes a ddrescue log and flags the duff sectors as "soft unreadable". That would mean that if you can recover 35 drives, provided no stripe has lost two sectors across two drives, you wouldn't lose any data.
If you want to try and write that utility? Or if you want to email me a ddrescue log with a bunch of failed sectors, I'll have a go at writing it myself :-)
Check out hdparm --make-bad-sector. You can get what you are describing by scripting that. It's marked very dangerous, but I guess if one has nothing to lose....
Phil
Quoted 4 lines · Martin Bosner
Is there a way tell mdadm to use sdk1 as a “active” device? And how can i tell the array that it should not try to recover sdm1 but set it active ? Is there any magic to force the state ? It might not be healthy for the normal use case but might be helpful for me.
No. There's no way to do that. Short of reading the kernel code to do hex editing on the superblock. And since the spare status is there due to an --add action, you can't trust that anything else there would be safe for --create --assume-clean. You would scramble these eggs even further.
Phil
Quoted 8 lines · Phil Turmel
Quoted 6 lines · Wols Lists
But there's a request on the linux wiki program for someone to write a utility program that takes a ddrescue log and flags the duff sectors as "soft unreadable". That would mean that if you can recover 35 drives, provided no stripe has lost two sectors across two drives, you wouldn't lose any data.
Quoted 8 lines
Quoted 3 lines
If you want to try and write that utility? Or if you want to email me a ddrescue log with a bunch of failed sectors, I'll have a go at writing it myself :-)
Check out hdparm --make-bad-sector. You can get what you are describing by scripting that. It's marked very dangerous, but I guess if one has nothing to lose....
Wol and I have tic-tacced on that a couple of times. He suggested the idea and I proved the viability of it by testing hdparm in a RAID to do exactly that. Neither of us has had a chance to stitch it all together, but the preliminary tests indicate that would do *exactly* what was required.
Given the hdparm commands are completely reversible and non-permanent, *and* they are being executed on a destination of a ddrescue, there is pretty much no risk (as long as the underlying glue code to be written gets the sector numbers and offsets right).
Regards,
Brad