Raid1 where Event Count off my 1 cannot assemble --force
Guys,
I have an older box that is a fax server where the Event Count for /dev/md1 is
off by 1, but the array cannot be reassembled with --assemble --force /dev/dm1 /dev/sda5 /dev/sdb5. Per the warnings in the wiki, I'm asking for help before I go attempt to recreate the array and screw something up. Here is the relevant information. The box is running openSuSE 11 (2.6.25) with mdraid 2.6.4. This box has run flawlessly for years.
I have 3 mdraid partitions on this box:
/dev/md0 sda1/sdb1 /boot /dev/md1 sda5/sdb5 / /dev/md2 sda7/sdb7 /home
After booting the 11.0 install dvd and booting Recovery Console, mdraid found
and assembled all arrays, md0 and md2 are fine, its is just md1 that is the problem:
# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda7[0] sdb7[1]
221929772 blocks super 1.0 [2/2] [UU]
bitmap: 0/424 pages [0KB], 256KB chunkmd1 : inactive sda5[0] sdb5[1]
41945504 blocks super 1.0
md0 : active raid1 sda1[0] sdb1[1]
104376 blocks super 1.0 [2/2] [UU]
bitmap: 0/7 pages [0KB], 8KB chunk The array information on disk for both disks (sda5/sdb5) shows the exact same
Update Time, (Tue Nov 19 15:28:38 2013) the only difference between the output is the checksums (both shown correct) and the Events : 148/149 The full output of mdadm --examine /dev/sd[ab]5 is here and in a (1.7 M screenshot) below:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
Name : 1 Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 41945504 (20.00 GiG 21.48 GB)
Array Size : 41945504 (20.00 GiG 21.48 GB)
Super Offset : 41945632 sectors
State : clean
Device UUID : e8c1c580:db4d853e:6fac1c8f:fb5399d7
Internal Bitmap : -81 sectors from superblock
Update Time : Tue Nov 19 15:28:38 2013
checksum : d37d1086 - correct
Events : 148
Array Slot : 0 (0,1)
Array State : Uu
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
Name : 1 Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 41945504 (20.00 GiG 21.48 GB)
Array Size : 41945504 (20.00 GiG 21.48 GB)
Super Offset : 41945632 sectors
State : clean
Device UUID : 6edfa3f8:c8c4316d:66c19315:5eda0911
Internal Bitmap : -81 sectors from superblock
Update Time : Tue Nov 19 15:28:38 2013
checksum : 39ef40a5 - correct
Events : 149
Array Slot : 1 (0,1)
Array State : uUhttp://www.3111skyline.com/dl/screenshots/suse/mdadm-examine.jpg (1.7 Meg)
I have read through https://raid.wiki.kernel.org/index.php/RAID_Recovery and I
can confirm that mdadm --stop /dev/md1, stops the array and removes the device for the information shown in cat /proc/mdstat. I have attempted to assemble and force to get the array running but I am left with the same Input/Output error.
What does it look like the next proper course of action should be? I am new to
triaging non-working raid arrays, so all I can do is read. The next step appears to be recreating the drives and hoping it all works. Am I at that "last resort" yet? or are there a few more tricks to try. Thank you in advance for any help you can give.
Signature · 1 line
David C. Rankin, J.D.,P.E.
Quoted 5 lines · David C. Rankin
Guys,
I have an older box that is a fax server where the Event Count for
/dev/md1is
off by 1, but the array cannot be reassembled with --assemble --force/dev/dm1/dev/sda5/dev/sdb5.
What are the messages displayed in "dmesg" when you try to use this command?
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 10 lines · Mikael Abrahamsson
Quoted 5 lines · David C. Rankin
Guys,
I have an older box that is a fax server where the Event Count for
/dev/md1is
off by 1, but the array cannot be reassembled with --assemble --force/dev/dm1/dev/sda5/dev/sdb5.What are the messages displayed in "dmesg" when you try to use this command?
Mikael,
Following the commands:
# mdadm --stop /dev/md1 # mdadm --assemble --force /dev/dm1 /dev/sd[ab]5
The messages captured in the logs are:
Rescue Kernel: md: md1: stopped.
Rescue Kernel: md: unbind<sda5>
Rescue Kernel: md: export_rdev(sda5)
Rescue Kernel: md: unbind<sdb5>
Rescue Kernel: md: export_rdev(sdb5)
Rescue Kernel: md: md1: stopped.
Rescue Kernel: md: md1 raid array is not clean -- starting background reconstruction Rescue Kernel: md: raid1: raid set md1 active with 2 out of 2 mirrors Rescue Kernel: md1: bitmap file is out of date (148 < 149) -- forcing full recovery Rescue Kernel: md1: bitmap file is out of date, doing full recovery Rescue Kernel: md1: bitmap initialisation failed: -5
Rescue Kernel: md1: failed to create bitmap (-5)
That's it for the log, then on the command line I have:
mdadm: failed to RUN_ARRAY /dev/md1: Input/Output error
What should I try next? Don't hesitate to ask if you need any additional
information, I'll provide whatever is necessary. Thanks.
Signature · 1 line
David C. Rankin, J.D.,P.E.
Quoted 35 lines · David C. Rankin
Quoted 10 lines · Mikael Abrahamsson
Quoted 5 lines · David C. Rankin
Guys,
I have an older box that is a fax server where the Event Count for
/dev/md1is
off by 1, but the array cannot be reassembled with --assemble --force/dev/dm1/dev/sda5/dev/sdb5.What are the messages displayed in "dmesg" when you try to use this command?
Mikael,
Following the commands:
# mdadm --stop /dev/md1 # mdadm --assemble --force /dev/dm1 /dev/sd[ab]5The messages captured in the logs are:
Rescue Kernel: md: md1: stopped.
Rescue Kernel: md: unbind<sda5>
Rescue Kernel: md:export_rdev(sda5)
Rescue Kernel: md: unbind<sdb5>
Rescue Kernel: md:export_rdev(sdb5)
Rescue Kernel: md: md1: stopped.
Rescue Kernel: md: md1 raid array is not clean -- starting background reconstruction Rescue Kernel: md: raid1: raid set md1 active with 2 out of 2 mirrors Rescue Kernel: md1: bitmap file is out of date (148 < 149) -- forcing full recovery Rescue Kernel: md1: bitmap file is out of date, doing full recovery Rescue Kernel: md1: bitmap initialisation failed: -5
Rescue Kernel: md1: failed to create bitmap (-5)
Quoted 7 lines
That's it for the log, then on the command line I have:
mdadm: failed to
RUN_ARRAY/dev/md1: Input/Output errorWhat should I try next? Don't hesitate to ask if you need any additional
information, I'll provide whatever is necessary. Thanks.
Here is additional information with --verbose given:
nemtemp:~ # cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda7[0] sdb7[1]
221929772 blocks super 1.0 [2/2] [UU]
bitmap: 0/424 pages [0KB], 256KB chunkmd1 : inactive sda5[0] sdb5[1]
41945504 blocks super 1.0
md0 : active raid1 sda1[0] sdb1[1]
104376 blocks super 1.0 [2/2] [UU]
bitmap: 0/7 pages [0KB], 8KB chunkunused devices: <none>
nemtemp:~ # mdadm --stop /dev/md1
mdadm: stopped /dev/md1
nemtemp:~ # cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda7[0] sdb7[1]
221929772 blocks super 1.0 [2/2] [UU]
bitmap: 0/424 pages [0KB], 256KB chunkmd0 : active raid1 sda1[0] sdb1[1]
104376 blocks super 1.0 [2/2] [UU]
bitmap: 0/7 pages [0KB], 8KB chunkunused devices: <none>
nemtemp:~ # mdadm --verbose --assemble --force /dev/md1 /dev/sd[ab]5
mdadm: looking for devices for /dev/md1
mdadm: /dev/sda5 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdb5 is identified as a member of /dev/md1, slot 1.
mdadm: added /dev/sdb5 to /dev/md1 as 1
mdadm: added /dev/sda5 to /dev/md1 as 0
mdadm: failed to RUN_ARRAY /dev/md1: Input/output error
The log from the start attempt:
Dec 9 00:16:11 Rescue kernel: md: md1 stopped. Dec 9 00:16:11 Rescue kernel: md: bind<sdb5> Dec 9 00:16:11 Rescue kernel: md: bind<sda5> Dec 9 00:16:11 Rescue kernel: md: md1: raid array is not clean -- starting background reconstruction Dec 9 00:16:11 Rescue kernel: raid1: raid set md1 active with 2 out of 2 mirrors Dec 9 00:16:11 Rescue kernel: md1: bitmap file is out of date (148 < 149) -- forcing full recovery Dec 9 00:16:11 Rescue kernel: md1: bitmap file is out of date, doing full recovery Dec 9 00:16:12 Rescue kernel: md1: bitmap initialisation failed: -5 Dec 9 00:16:12 Rescue kernel: md1: failed to create bitmap (-5) Dec 9 00:16:12 Rescue kernel: md: pers->run() failed ...
nemtemp:~ # cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda7[0] sdb7[1]
221929772 blocks super 1.0 [2/2] [UU]
bitmap: 0/424 pages [0KB], 256KB chunkmd1 : inactive sda5[0] sdb5[1]
41945504 blocks super 1.0
md0 : active raid1 sda1[0] sdb1[1]
104376 blocks super 1.0 [2/2] [UU]
bitmap: 0/7 pages [0KB], 8KB chunkunused devices: <none>
I'm not sure how to proceed safely from here. Is there anything else I should
try before attempting to --create the array again? If we do create the array with 1 drive and "missing", should I then use --add or --re-add to add the other drive? Also, since /dev/sda5 shows Events: 148 and /dev/sdb5 shows Events: 149, should I choose /dev/sdb5 as the one to preserve and let "missing" take the place of /dev/sda5? If so, then does the following create statement look correct:
mdadm --create --verbose --level=1 --metadata=1.0 --raid-devices=2 \/dev/md1 /dev/sdb5 missing
Should I also use --force?
If attempting to assemble with "missing" and the create command gives problems
due to the unused device still having the same minor-number, is it better to --zero-superblock the on the device not included as "missing" or is it better to just unplug it and preserve the superblock data in case it is needed?
Sorry for all the questions, but I just want to make sure I don't do something
to compromise the data. With the information for both drives looking good with --examine, the (Update Time : Tue Nov 19 15:28:38 2013) being identical, and the Events being off by only 1, I can't see a reason the drives should not just assemble and run as it is. What say the experts?
Here is the --detail and --examine information for the drives for completeness:
nemtemp:~ # mdadm --detail /dev/md1/dev/md1:
Version : 01.00.03
Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Used Dev Size : 20972752 (20.00 GiB 21.48 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Nov 19 15:28:38 2013
State : active, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : 1
UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
Events : 148
Number Major Minor RaidDevice State
0 8 5 0 active sync /dev/sda5
1 8 21 1 active sync /dev/sdb5
nemtemp:/ # mdadm -E /dev/sda5/dev/sda5:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
Name : 1 Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 41945504 (20.00 GiB 21.48 GB)
Array Size : 41945504 (20.00 GiB 21.48 GB)
Super Offset : 41945632 sectors
State : clean
Device UUID : e0c1c580:db4d853e:6fac1c8f:fb5399d7
Internal Bitmap : -81 sectors from superblock
Update Time : Tue Nov 19 15:28:38 2013
Checksum : d37d1086 - correct
Events : 148
Array Slot : 0 (0, 1)Array State : Uu
nemtemp:/ # mdadm -E /dev/sdb5/dev/sdb5:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
Name : 1 Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 41945504 (20.00 GiB 21.48 GB)
Array Size : 41945504 (20.00 GiB 21.48 GB)
Super Offset : 41945632 sectors
State : active
Device UUID : 6edfa3f8:c8c4316d:66c19315:5eda0911
Internal Bitmap : -81 sectors from superblock
Update Time : Tue Nov 19 15:28:38 2013
Checksum : 39ef40a5 - correct
Events : 149
Array Slot : 1 (0, 1)Array State : uU
Signature · 1 line
David C. Rankin, J.D.,P.E.
Quoted 10 lines · David C. Rankin
Quoted 8 lines · David C. Rankin
Quoted 7 lines · Mikael Abrahamsson
Quoted 5 lines · David C. Rankin
Guys,
I have an older box that is a fax server where the Event Count for
/dev/md1is
off by 1, but the array cannot be reassembled with --assemble --force/dev/dm1/dev/sda5/dev/sdb5.Here is additional information with --verbose given:
Have you tried this:
mdadm --verbose --assemble /dev/md1 /dev/sdb5
mdadm --manage /dev/md1 --run
Being raid1, you should be able to use only a single device....
BTW, chances to recover your data should be exceptional, as long as you don't do anything too silly. You should even be able to mount the device directly (read-only):
mount -o ro /dev/sdb5 /mnt
(Depending on the content is a filesystem).
Then you can just backup the data, create a new array, and restore the data. Depending on data and size this might even be a better option...
Quoted 71 lines
Here is the --detail and --examine information for the drives for completeness:
nemtemp:~ # mdadm --detail
/dev/md1/dev/md1:Version : 01.00.03Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Used Dev Size : 20972752 (20.00 GiB 21.48 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 1
Persistence : Superblock is persistentUpdate Time : Tue Nov 19 15:28:38 2013
State : active, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0Name : 1
UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
Events : 148Number Major Minor RaidDevice State
0 8 5 0 active sync/dev/sda5
1 8 21 1 active sync/dev/sdb5nemtemp:/ # mdadm -E
/dev/sda5/dev/sda5:Magic : a92b4efc Version : 1.0 Feature Map : 0x1 Array UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25 Name : 1Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Raid Devices : 2Avail Dev Size : 41945504 (20.00 GiB 21.48 GB)
Array Size : 41945504 (20.00 GiB 21.48 GB) Super Offset : 41945632 sectors State : clean Device UUID : e0c1c580:db4d853e:6fac1c8f:fb5399d7Internal Bitmap : -81 sectors from superblock
Update Time : Tue Nov 19 15:28:38 2013 Checksum : d37d1086 - correct Events : 148 Array Slot : 0 (0, 1) Array State : Uunemtemp:/ # mdadm -E
/dev/sdb5/dev/sdb5:Magic : a92b4efc Version : 1.0 Feature Map : 0x1 Array UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25 Name : 1Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Raid Devices : 2Avail Dev Size : 41945504 (20.00 GiB 21.48 GB)
Array Size : 41945504 (20.00 GiB 21.48 GB) Super Offset : 41945632 sectors State : active Device UUID : 6edfa3f8:c8c4316d:66c19315:5eda0911
Quoted 8 lines
Internal Bitmap : -81 sectors from superblock
Update Time : Tue Nov 19 15:28:38 2013 Checksum : 39ef40a5 - correct Events : 149 Array Slot : 1 (0, 1) Array State : uU
BTW, the bitmap location looks.... strange...
Regards,
Adam
Signature · 1 line
Adam Goryachev Website Managers www.websitemanagers.com.au
Quoted 208 lines · On Sun, 08 Dec 2013 18:38:58 -0600 "David C. Rankin"…
Quoted 35 lines · David C. Rankin
Quoted 10 lines · Mikael Abrahamsson
Quoted 5 lines · David C. Rankin
Guys,
I have an older box that is a fax server where the Event Count for
/dev/md1is
off by 1, but the array cannot be reassembled with --assemble --force/dev/dm1/dev/sda5/dev/sdb5.What are the messages displayed in "dmesg" when you try to use this command?
Mikael,
Following the commands:
# mdadm --stop /dev/md1 # mdadm --assemble --force /dev/dm1 /dev/sd[ab]5The messages captured in the logs are:
Rescue Kernel: md: md1: stopped.
Rescue Kernel: md: unbind<sda5>
Rescue Kernel: md:export_rdev(sda5)
Rescue Kernel: md: unbind<sdb5>
Rescue Kernel: md:export_rdev(sdb5)
Rescue Kernel: md: md1: stopped.
Rescue Kernel: md: md1 raid array is not clean -- starting background reconstruction Rescue Kernel: md: raid1: raid set md1 active with 2 out of 2 mirrors Rescue Kernel: md1: bitmap file is out of date (148 < 149) -- forcing full recovery Rescue Kernel: md1: bitmap file is out of date, doing full recovery Rescue Kernel: md1: bitmap initialisation failed: -5
Rescue Kernel: md1: failed to create bitmap (-5)Quoted 7 lines
That's it for the log, then on the command line I have:
mdadm: failed to
RUN_ARRAY/dev/md1: Input/Output errorWhat should I try next? Don't hesitate to ask if you need any additional
information, I'll provide whatever is necessary. Thanks.Here is additional information with --verbose given:
nemtemp:~ # cat
/proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda7[0] sdb7[1]221929772 blocks super 1.0 [2/2] [UU] bitmap: 0/424 pages [0KB], 256KB chunkmd1 : inactive sda5[0] sdb5[1]
41945504 blocks super 1.0md0 : active raid1 sda1[0] sdb1[1]
104376 blocks super 1.0 [2/2] [UU] bitmap: 0/7 pages [0KB], 8KB chunkunused devices: <none>
nemtemp:~ # mdadm --stop
/dev/md1
mdadm: stopped/dev/md1nemtemp:~ # cat
/proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda7[0] sdb7[1]221929772 blocks super 1.0 [2/2] [UU] bitmap: 0/424 pages [0KB], 256KB chunkmd0 : active raid1 sda1[0] sdb1[1]
104376 blocks super 1.0 [2/2] [UU] bitmap: 0/7 pages [0KB], 8KB chunkunused devices: <none>
nemtemp:~ # mdadm --verbose --assemble --force
/dev/md1/dev/sd[ab]5
mdadm: looking for devices for/dev/md1
mdadm:/dev/sda5is identified as a member of/dev/md1, slot 0.
mdadm:/dev/sdb5is identified as a member of/dev/md1, slot 1.
mdadm: added/dev/sdb5to/dev/md1as 1
mdadm: added/dev/sda5to/dev/md1as 0
mdadm: failed toRUN_ARRAY/dev/md1: Input/output errorThe log from the start attempt:
Dec 9 00:16:11 Rescue kernel: md: md1 stopped. Dec 9 00:16:11 Rescue kernel: md: bind<sdb5> Dec 9 00:16:11 Rescue kernel: md: bind<sda5> Dec 9 00:16:11 Rescue kernel: md: md1: raid array is not clean -- starting background reconstruction Dec 9 00:16:11 Rescue kernel: raid1: raid set md1 active with 2 out of 2 mirrors Dec 9 00:16:11 Rescue kernel: md1: bitmap file is out of date (148 < 149) -- forcing full recovery Dec 9 00:16:11 Rescue kernel: md1: bitmap file is out of date, doing full recovery Dec 9 00:16:12 Rescue kernel: md1: bitmap initialisation failed: -5 Dec 9 00:16:12 Rescue kernel: md1: failed to create bitmap (-5) Dec 9 00:16:12 Rescue kernel: md: pers->run() failed ...nemtemp:~ # cat
/proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda7[0] sdb7[1]221929772 blocks super 1.0 [2/2] [UU] bitmap: 0/424 pages [0KB], 256KB chunkmd1 : inactive sda5[0] sdb5[1]
41945504 blocks super 1.0md0 : active raid1 sda1[0] sdb1[1]
104376 blocks super 1.0 [2/2] [UU] bitmap: 0/7 pages [0KB], 8KB chunkunused devices: <none>
I'm not sure how to proceed safely from here. Is there anything else I should
try before attempting to --create the array again? If we do create the array with 1 drive and "missing", should I then use --add or --re-add to add the other drive? Also, since/dev/sda5shows Events: 148 and/dev/sdb5shows Events: 149, should I choose/dev/sdb5as the one to preserve and let "missing" take the place of/dev/sda5? If so, then does the following create statement look correct:mdadm --create --verbose --level=1 --metadata=1.0 --raid-devices=2 \
/dev/md1/dev/sdb5missingShould I also use --force?
If attempting to assemble with "missing" and the create command gives problems
due to the unused device still having the same minor-number, is it better to --zero-superblock the on the device not included as "missing" or is it better to just unplug it and preserve the superblock data in case it is needed?Sorry for all the questions, but I just want to make sure I don't do something
to compromise the data. With the information for both drives looking good with --examine, the (Update Time : Tue Nov 19 15:28:38 2013) being identical, and the Events being off by only 1, I can't see a reason the drives should not just assemble and run as it is. What say the experts?Here is the --detail and --examine information for the drives for completeness:
nemtemp:~ # mdadm --detail
/dev/md1/dev/md1:Version : 01.00.03Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Used Dev Size : 20972752 (20.00 GiB 21.48 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 1
Persistence : Superblock is persistentUpdate Time : Tue Nov 19 15:28:38 2013
State : active, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0Name : 1
UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25
Events : 148Number Major Minor RaidDevice State
0 8 5 0 active sync/dev/sda5
1 8 21 1 active sync/dev/sdb5nemtemp:/ # mdadm -E
/dev/sda5/dev/sda5:Magic : a92b4efc Version : 1.0 Feature Map : 0x1 Array UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25 Name : 1Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Raid Devices : 2Avail Dev Size : 41945504 (20.00 GiB 21.48 GB)
Array Size : 41945504 (20.00 GiB 21.48 GB)Super Offset : 41945632 sectors
State : clean
Device UUID : e0c1c580:db4d853e:6fac1c8f:fb5399d7Internal Bitmap : -81 sectors from superblock
Update Time : Tue Nov 19 15:28:38 2013 Checksum : d37d1086 - correct Events : 148 Array Slot : 0 (0, 1)Array State : Uu
nemtemp:/ # mdadm -E
/dev/sdb5/dev/sdb5:Magic : a92b4efc Version : 1.0 Feature Map : 0x1 Array UUID : e45cfbeb:77c2b93b:43d3d214:390d0f25 Name : 1Creation Time : Thu Aug 21 06:43:22 2008
Raid Level : raid1
Raid Devices : 2Avail Dev Size : 41945504 (20.00 GiB 21.48 GB)
Array Size : 41945504 (20.00 GiB 21.48 GB)Super Offset : 41945632 sectors
State : active
Device UUID : 6edfa3f8:c8c4316d:66c19315:5eda0911
Quoted 11 lines
Internal Bitmap : -81 sectors from superblock
Update Time : Tue Nov 19 15:28:38 2013 Checksum : 39ef40a5 - correct Events : 149 Array Slot : 1 (0, 1)Array State : uU
What version of mdadm do you have? It looks like it should be cleverer than it is.
What if you add "--update=no-bitmap" to the --assemble line? As the bitmap seems to be causing problem, ignoring it might help.
NeilBrown
Quoted 8 lines · Adam Goryachev
Have you tried this:
mdadm --verbose --assemble
/dev/md1/dev/sdb5
mdadm --manage/dev/md1--runBeing raid1, you should be able to use only a single device....
Quoted 8 lines
BTW, chances to recover your data should be exceptional, as long as you don't do anything too silly. You should even be able to mount the device directly (read-only):
mount -o ro/dev/sdb5/mnt(Depending on the content is a filesystem).
Then you can just backup the data, create a new array, and restore the data. Depending on data and size this might even be a better option...
Adam,
Thank you for your suggestions. Here is the output attempting what you
suggested. The version is old (2.6.4), it is on the openSuSE install DVD for 11.0:
nemtemp:/mnt # mdadm --verbose --assemble /dev/md1 /dev/sdb5
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb5 is identified as a member of /dev/md1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md1
mdadm: added /dev/sdb5 to /dev/md1 as 1
mdadm: /dev/md1 assembled from 1 drive - need all 2 to start it (use --run to insist).
nemtemp:/mnt # cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda7[0] sdb7[1]
221929772 blocks super 1.0 [2/2] [UU]
bitmap: 0/424 pages [0KB], 256KB chunkmd1 : inactive sdb5[1](S)
20972752 blocks super 1.0
md0 : active raid1 sda1[0] sdb1[1]
104376 blocks super 1.0 [2/2] [UU]
bitmap: 0/7 pages [0KB], 8KB chunkunused devices: <none>
nemtemp:/mnt # mdadm --run --verbose /dev/md1
mdadm: failed to run array /dev/md1: Input/output error nemtemp:/mnt # tail /var/log/messages
Dec 9 02:30:20 Rescue kernel: raid1: raid set md1 active with 1 out of 2 mirrors Dec 9 02:30:20 Rescue kernel: md1: bitmap file is out of date (148 < 149) -- forcing full recovery Dec 9 02:30:20 Rescue kernel: md1: bitmap file is out of date, doing full recovery Dec 9 02:30:20 Rescue kernel: md1: bitmap initialisation failed: -5 Dec 9 02:30:20 Rescue kernel: md1: failed to create bitmap (-5) Dec 9 02:30:20 Rescue kernel: md: pers->run() failed ...
I could boot with a newer recovery disk and see if a newer version of mdadm
would do things differently. I was using the original just to make sure I didn't outsmart myself using a never version of mdadm than I would be running with after repairing /dev/md1
Hooray! I can mount the thing as -t ext3:
mdadm: stopped /dev/md1
nemtemp:/mnt # mount -o ro /dev/sdb5 /mnt/sdb/
mount: unknown filesystem type 'linux_raid_member' nemtemp:/mnt # mount -t ext3 -o ro /dev/sdb5 /mnt/sdb/ nemtemp:/mnt # l sdb
total 116
drwxr-xr-x 21 root root 4096 2013-01-25 17:06 ./
drwxr-xr-x 7 root root 140 2013-12-08 06:38 ../ drwxr-xr-x 2 root root 4096 2010-12-05 06:43 bin/ drwxr-xr-x 2 root root 4096 2008-08-21 06:48 boot/ drwxr-xr-x 2 root root 4096 2008-08-22 01:54 data/ drwxr-xr-x 5 root root 4096 2008-08-21 06:48 dev/ drwxr-xr-x 129 root root 12288 2013-11-16 13:23 etc/ drwxr-xr-x 2 root root 4096 2008-08-21 06:48 home/ drwxr-xr-x 14 root root 12288 2011-01-14 20:13 lib/ drwx------ 2 root root 16384 2008-08-21 06:43 lost+found/ drwxr-xr-x 2 root root 4096 2009-07-28 22:39 media/ drwxr-xr-x 8 root root 4096 2010-12-21 18:05 mnt/ drwxr-xr-x 5 root root 4096 2008-07-03 21:16 opt/ drwxr-xr-x 3 root root 4096 2008-08-21 06:48 proc/ drwx------ 24 root root 4096 2013-10-01 20:58 root/ drwxr-xr-x 3 root root 12288 2010-12-27 23:15 sbin/ drwxr-xr-x 4 root root 4096 2008-09-11 07:26 srv/ drwxr-xr-x 3 root root 4096 2008-08-21 06:48 sys/ drwxrwxrwt 7 root root 4096 2013-11-19 15:15 tmp/ drwxr-xr-x 12 root root 4096 2010-01-24 01:41 usr/ drwxr-xr-x 15 root root 4096 2009-07-02 06:37 var/
Now since I can mount it, how in the heck do I get the raid put back together.
Seems really simple, but I'm stuck... Try with a newer mdadm?
Quoted 1 line
BTW, the bitmap location looks.... strange...
I thought so too, but checking the other arrays, /dev/md2 has a negative
number as well:
nemtemp:/mnt # mdadm -E /dev/sda7/dev/sda7:
<snip>
Internal Bitmap : -213 sectors from superblock
Update Time : Mon Dec 9 02:14:18 2013
Signature · 1 line
David C. Rankin, J.D.,P.E.
Quoted 17 lines · David C. Rankin
Quoted 8 lines · Adam Goryachev
Have you tried this:
mdadm --verbose --assemble
/dev/md1/dev/sdb5
mdadm --manage/dev/md1--runBeing raid1, you should be able to use only a single device....
Quoted 8 lines
BTW, chances to recover your data should be exceptional, as long as you don't do anything too silly. You should even be able to mount the device directly (read-only):
mount -o ro/dev/sdb5/mnt(Depending on the content is a filesystem).
Then you can just backup the data, create a new array, and restore the data. Depending on data and size this might even be a better option...
Quoted 8 lines
nemtemp:/mnt # mount -o ro
/dev/sdb5/mnt/sdb/
mount: unknown filesystem type 'linux_raid_member' nemtemp:/mnt # mount -t ext3 -o ro/dev/sdb5/mnt/sdb/Now since I can mount it, how in the heck do I get the raid put back together.
Seems really simple, but I'm stuck... Try with a newer mdadm?
Probably the best option is to follow Neil's advise to use mdadm from git....
The alternative as I mentioned is to backup the data, re-create the raid + filesystem, and then restore the data.
Quoted 9 lines
Quoted 1 line
BTW, the bitmap location looks.... strange...
I thought so too, but checking the other arrays,
/dev/md2has a negative
number as well:nemtemp:/mnt # mdadm -E
/dev/sda7/dev/sda7:
<snip>
Internal Bitmap : -213 sectors from superblockUpdate Time : Mon Dec 9 02:14:18 2013
It looks strange when I first saw it, but now that I think about it, it is probably right (correct) since 1.0 metadata is at the very end of the drive, so the bitmap is probably before the metadata, hence negative offset.
Signature · 1 line
Adam Goryachev Website Managers www.websitemanagers.com.au
Quoted 8 lines · Adam Goryachev
Probably the best option is to follow Neil's advise to use mdadm from git....
The alternative as I mentioned is to backup the data, re-create the raid + filesystem, and then restore the data.
Quoted 4 lines
Quoted 1 line
BTW, the bitmap location looks.... strange...
I thought so too, but checking the other arrays,
/dev/md2has a negative
number as well:
Quoted 9 lines
Quoted 5 lines
nemtemp:/mnt # mdadm -E
/dev/sda7/dev/sda7:
<snip>
Internal Bitmap : -213 sectors from superblockUpdate Time : Mon Dec 9 02:14:18 2013It looks strange when I first saw it, but now that I think about it, it is probably right (correct) since 1.0 metadata is at the very end of the drive, so the bitmap is probably before the metadata, hence negative offset.
I have an install cd with mdadm 3.3.2 on it, I'll give that a go and see what it does with the array. The partition is only 20G, so I can just copy it to a new drive to backup. It almost seems like I should be able to change the Events number with a low level tool on one drive and see if that would fix the problem.
I suspect the problem is with the older mdadm. Searching, there were several posts very similar to mine in the 2008/2009 time frame. The older mdadm probably does not handle recovery very well.
If I do get the drive to assemble/run under mdadm 3.3.2, then is there anything I need to do before shutting down the box to insure it will work again under 2.6 so I can at least boot it before updating mdadm? If it assembles under 3.3.2, then I should be able to assemble/run it under 2.6.4 since the Event count would match -- right?
Signature · 1 line
David C. Rankin, J.D.,P.E.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 7 lines · NeilBrown
What version of mdadm do you have? It looks like it should be cleverer than it is.
What if you add "--update=no-bitmap" to the --assemble line? As the bitmap seems to be causing problem, ignoring it might help.
NeilBrown
I tried that too Neil, it said:
nemtemp:/mnt # mdadm --verbose --assemble --force --update=no-bitmap /dev/md1 /dev/sd[ab]5
mdadm: '--update=no-bitmap' is invalid. Valid --update options are:
'sparc2.2', 'super-minor', 'uuid', 'name', 'resync', 'summaries', 'homehost', 'byteorder', 'devicesize'.
Like in my reply to Adam, I think the age of mdadm may be the issue. The openSuSE 11.0 install DVD is pretty old ;-) It just may not be handling the
- --assemble correctly given the Event count difference, and there doesn't seem to be a way to just tell it:
"The data is correct on both disks, just assemble and run it and be quiet!"
I'll try booting an Arch install CD with 3.3.2 on it and report back. Thank you and Adam for your help.
If it does assemble with mdadm 3.3.2, anything I need to do on the assembled array to make sure it stays that way? fsck? Any way to tell it to make sure it is in sync so it will boot under the openSuSE 11.0 version of mdadm?
- --
David C. Rankin, J.D.,P.E.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlKlRtsACgkQZMpuZ8CyrcjDdACeIEoiMdNGQw0IEqg0LlgJ8f2t vHwAnjLKHnTP1FpEpXwjRZxu+hoAntQV
=iTBQ
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 25 lines · On Sun, 08 Dec 2013 22:28:11 -0600 "David C. Rankin"…
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 7 lines · NeilBrown
What version of mdadm do you have? It looks like it should be cleverer than it is.
What if you add "--update=no-bitmap" to the --assemble line? As the bitmap seems to be causing problem, ignoring it might help.
NeilBrown
I tried that too Neil, it said:
nemtemp:/mnt # mdadm --verbose --assemble --force --update=no-bitmap
/dev/md1/dev/sd[ab]5
mdadm: '--update=no-bitmap' is invalid. Valid --update options are:
'sparc2.2', 'super-minor', 'uuid', 'name', 'resync', 'summaries', 'homehost', 'byteorder', 'devicesize'.Like in my reply to Adam, I think the age of mdadm may be the issue. The openSuSE 11.0 install DVD is pretty old ;-) It just may not be handling the
- --assemble correctly given the Event count difference, and there doesn't seem to be a way to just tell it:
Quoted 9 lines
"The data is correct on both disks, just assemble and run it and be quiet!"
I'll try booting an Arch install CD with 3.3.2 on it and report back. Thank you and Adam for your help.
If it does assemble with mdadm 3.3.2, anything I need to do on the assembled array to make sure it stays that way? fsck? Any way to tell it to make sure it is in sync so it will boot under the openSuSE 11.0 version of mdadm?
Once you get any kernel to successfully assemble the array, any other kernel/mdadm should be able to as well.
NeilBrown
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIVAwUBUqVLLznsnt1WYoG5AQJ5mg//UkmhkCAtbiCf7z+1X9i/HsoROqH3Dhn6 ii2GEkWcb+IL7tYJ3zp0uzXPJOAuU10SSF7Icrobp9ABnWsCHefEXgGt4kRB7dG8 VkIrYIEArmSkLXkRlSP76IWi2A/AZzhBC3/8QPVXVDBy2I93YHwbkIttzLJURc8R /gtZRX7Bli4FgdNQuRj9K3KLc7Rar+CG1PmzLSInv3bXhZKM2JjlT2Wz4C0niBrf 0topMmjTrVESf8qT2QlOAvhN8BPNtWR95/H6W+v0nKU1KnyOuo6hwolqOFmspyJv Hrn2PFdNKmYFfQ+rFOMZQk4mxII9OxLp5bidE7r4jedzyhQi6QKDLcXZc550irxI e2vGu1Yg+V8d6dcRYgePChDIJn0c13iYgxZHxOJ4I1uhwC9dHNwgN8v377nkfZi+ wBml7Do2oynzImeCC7FVBnYrjmJm1M5djUqB12dIvQZySfI/SUMIVLPNyupiaSvb xvZ/QzoJaowYo1a0lt5EmWAVD5FZ/YTjiffxCdRr/P1rVWfc7Jx2KE/i42heXo8e zBU51hBrWI3JVLYMQr4/Rc1NYXtE51Cq0QzzM5dlnb+ya1m2boscYyUCZHjMGsJg 9JUnT6u7U8a676SZqCCUudIVPFxIR+xyhgpw0AmHcpfMdJ18ic/NmWXIQR7j+jka e5LiyQVu5nw=
=nGpK
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 4 lines · NeilBrown
Once you get any kernel to successfully assemble the array, any other kernel/mdadm should be able to as well.
NeilBrown
After 2 days pulling my hair out, the answer was a simple as popping in the Arch install CD and rebooting!!
After it booted, I checked cat /proc/mdstat and /dev/md1 (md125) on Arch is active and re-syncing. Perfect.
That was a whole lot of work to find out that it was an old mdadm causing all the problems -- live and learn...
Perhaps we should add "get the latest release of mdadm" to the Raid Recovery page with the explanation that sometimes older versions of mdadm just do not know how to handle simple issues that arise, such as differing Event counts. Using the latest mdadm can solve those problems immediately.
- --
David C. Rankin, J.D.,P.E.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlKlUxsACgkQZMpuZ8CyrchuWgCfR+Lv97xnyE75Yp3iTCfEq7aC +f8An2d6+MOHw0Bki+kg5IvOIxC7yYsr
=T4Hs
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 15 lines · On Sun, 08 Dec 2013 23:20:27 -0600 "David C. Rankin"…
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 4 lines · NeilBrown
Once you get any kernel to successfully assemble the array, any other kernel/mdadm should be able to as well.
NeilBrown
After 2 days pulling my hair out, the answer was a simple as popping in the Arch install CD and rebooting!!
After it booted, I checked cat
/proc/mdstatand/dev/md1(md125) on Arch is active and re-syncing. Perfect.
Quoted 7 lines
That was a whole lot of work to find out that it was an old mdadm causing all the problems -- live and learn...
Perhaps we should add "get the latest release of mdadm" to the Raid Recovery page with the explanation that sometimes older versions of mdadm just do not know how to handle simple issues that arise, such as differing Event counts. Using the latest mdadm can solve those problems immediately.
Glad that it is now working for you!
Isn't "Make sure you have latest release" always the second thing to check (after checking that you are actually using the command properly:-)??
But yes, it doesn't hurt to add it to a wiki page somewhere.
NeilBrown
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIVAwUBUqVXtTnsnt1WYoG5AQLG/BAApvOlBcyQ199ZiRoh2IYz3IgZg9ovmoTn zdWNN5frwFx4mUmq1+eUpKpn7XQ3XADBIbF82/1vSa0GQ2SfFkpUbJQycaxXgky2 Bf6JtQ6XFBmbgu1XYETFJfBRV9m9Gk10bKsGeNtqvRpfelqJ4ucIyFMPwktIXnON Xe6ailKNr62fRCGmE3fxjWujMgiJjy/Q3tTbAOZvZ+8TLFbZSdFRlG5gIQSqVZP2 O3ZGkvgFYVseTIOMVzbeQg6VxRdTaWbEZ3xpyIa47/l6/h0ihnldk4NUC3bwDT1D lxFLkKcgFgrPargjQ0wsSvx7/QC4pbk0cuKOk8Y1Y7O/eJny028v0tEzarXuZbgp eDYtXht6lESsNRiyAkTqbeu5VydWYsGRUipmg0L4QtmP3TPjfWGYHZZxU+tPBfns q2QvaUQpLIjFpr+y3C2k75xKIuoZEwReNMSJGvI/DB5CD8WqJIVqIy8LsIcHujxP LAm9OCRnFNylVedEcAeHUju1Qyir0vPcs3wRrEJIv0yPR6ObfObFPtpzS0zYyCZm YcmI63a6DS2q7WcGluYSwOgMu1tmmdU2/sifWE7+JZh6PI56Ilt3+RRfQRObzyk8 8/7o9O4VSVB6BSEyFqthuC16N/wiSSACPSqP2Li1IDJ/tPUv5WPc9FCKG6BYlkvF SFLL7BeZygc=
=KkUZ
-----END PGP SIGNATURE-----
Quoted 1 line · NeilBrown
But yes, it doesn't hurt to add it to a wiki page somewhere.
I added this text to the wiki:
"There are bugs in older versions of mdadm, and a lot of "stable" operating system releases ship with really old mdadm versions. Recent versions are 3.2.x and 3.3.x. It's advisable if you run into problems assembling your raid to upgrade to the latest git version of mdadm. If you can get the raid to successfully assemble and recover with the git version, then it's fine to use your old mdadm version again for normal system operation. Newer mdadm doesn't make any changes to the array that isn't backwards compatible."
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 11 lines · Mikael Abrahamsson
Quoted 1 line
But yes, it doesn't hurt to add it to a wiki page somewhere.
I added this text to the wiki:
"There are bugs in older versions of mdadm, and a lot of "stable" operating system releases ship with really old mdadm versions. Recent versions are 3.2.x and 3.3.x. It's advisable if you run into problems assembling your raid to upgrade to the latest git version of mdadm. If you can get the raid to successfully assemble and recover with the git version, then it's fine to use your old mdadm version again for normal system operation. Newer mdadm doesn't make any changes to the array that isn't backwards compatible."
That ought to help the dummies out...
As follow-up and confirmation of the help statement. I can confirm that after
repairing my arrays under mdadm 3.3.2, I was able to reboot and have the arrays assemble and run with mdadm 2.6.4 without any issue. (good thing since the problem was on the / partitions...)
Now I just have to back-port mdadm-master to build and run on openSuSE 11.0
when the closest thing I can find is an 11.1 src.rpm. (should be close enough)
Thank you again Neil and Mikael for all the help.
Signature · 1 line
David C. Rankin, J.D.,P.E.