Skip to contentSkip to conversation

New raid level suggestion.

DISCUSSION Rogier Wolff 8 participants 16 messages Dec 30, 2010 – Jan 4, 2011 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi,

A friend has a webserver. He has 4 drive bays and due to previous problems he's not content to have 3 or 4 drives in a raid5 configuration, but he wants a "hot spare" so that when it takes him a week to find a new drive and some time to drive to the hosting company, he isn't susceptible to a second drive crashing in the meantime.

So in principle he'll build a 3-drive RAID5 with a hot spare....

Now we've been told that raid5 performs badly for the workload that is expected. It would be much better to run the system in RAID10. However if he'd switch to RAID10, after a single drive failure he has a window of about a week where he has a 33% chance of a second drive failure being "fatal".

So I was thinking.... He's resigned himself to a configuration where he pays for 4x the disk space and only gets 2x the available space.

So he could run his array in RAID10 mode, however when a drive fails, a fallback to raid5 would be in order. In this case, after the resync a single-drive-failure tolerance is again obtained.

In practise scaling down to raid5 is not easy/possible. RAID4 however should be doable.

In fact this can almost be implemented entirely in userspace. Just remove the mirror drive from the underlying raid0, and reinitialize as raid4. If you do this correctly the data will still be there....

Although doing this with an active filesystem running on these drives is probably impossible due to "device is in use" error messages....

So: Has anybody tried this before?
Can this be implemented without kernel support?
Anybody feel like implementing this?

Roger.

Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
#1 lore raw
Quoted 32 lines · Rogier Wolff

Hi,

A friend has a webserver. He has 4 drive bays and due to previous problems he's not content to have 3 or 4 drives in a raid5 configuration, but he wants a "hot spare" so that when it takes him a week to find a new drive and some time to drive to the hosting company, he isn't susceptible to a second drive crashing in the meantime.

So in principle he'll build a 3-drive RAID5 with a hot spare....

Now we've been told that raid5 performs badly for the workload that is expected. It would be much better to run the system in RAID10. However if he'd switch to RAID10, after a single drive failure he has a window of about a week where he has a 33% chance of a second drive failure being "fatal".

So I was thinking.... He's resigned himself to a configuration where he pays for 4x the disk space and only gets 2x the available space.

So he could run his array in RAID10 mode, however when a drive fails, a fallback to raid5 would be in order. In this case, after the resync a single-drive-failure tolerance is again obtained.

In practise scaling down to raid5 is not easy/possible. RAID4 however should be doable.

In fact this can almost be implemented entirely in userspace. Just remove the mirror drive from the underlying raid0, and reinitialize as raid4. If you do this correctly the data will still be there....

Quoted 9 lines

Although doing this with an active filesystem running on these drives is probably impossible due to "device is in use" error messages....

So: Has anybody tried this before?
Can this be implemented without kernel support?
Anybody feel like implementing this?

Roger.

Maybe I'm not quite understanding right, however you can easily do RAID6 with 4 drives. That will give you two redundant, effectively give you RAID5 if I drive fails, and save buttloads of messing around...

Signature · 6 lines
Steven Haigh

Email: [email protected]
Web: http://www.crc.id.au
Phone: (03) 9001 6090 - 0412 935 897
Fax: (03) 8338 0299
#2 lore raw
Quoted 3 lines · Steven Haigh

Maybe I'm not quite understanding right, however you can easily do RAID6 with 4 drives. That will give you two redundant, effectively give you RAID5 if I drive fails, and save buttloads of messing around...

Steven, My friend has a server where the drives take up to a third of a second to respond. When asking for help, everybody pounced on us:
- NEVER use raid5 for a server doing small-file-io like a mailserver. (always use RAID10).

So apparently RAID5 (and by extension RAID6) is not an option for some systems.

I'm willing to tolerate the RAID4 situation during the time that it takes me to replace the drive.

Roger.

Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
#14 lore raw
In reply to Rogier Wolff
Quoted 32 lines · On Thu, 30 Dec 2010 09:23:56 +0100 Rogier Wolff <[email protected]> wrote:

Hi,

A friend has a webserver. He has 4 drive bays and due to previous problems he's not content to have 3 or 4 drives in a raid5 configuration, but he wants a "hot spare" so that when it takes him a week to find a new drive and some time to drive to the hosting company, he isn't susceptible to a second drive crashing in the meantime.

So in principle he'll build a 3-drive RAID5 with a hot spare....

Now we've been told that raid5 performs badly for the workload that is expected. It would be much better to run the system in RAID10. However if he'd switch to RAID10, after a single drive failure he has a window of about a week where he has a 33% chance of a second drive failure being "fatal".

So I was thinking.... He's resigned himself to a configuration where he pays for 4x the disk space and only gets 2x the available space.

So he could run his array in RAID10 mode, however when a drive fails, a fallback to raid5 would be in order. In this case, after the resync a single-drive-failure tolerance is again obtained.

In practise scaling down to raid5 is not easy/possible. RAID4 however should be doable.

In fact this can almost be implemented entirely in userspace. Just remove the mirror drive from the underlying raid0, and reinitialize as raid4. If you do this correctly the data will still be there....

Quoted 9 lines

Although doing this with an active filesystem running on these drives is probably impossible due to "device is in use" error messages....

So: Has anybody tried this before?
Can this be implemented without kernel support?
Anybody feel like implementing this?

Roger.

The kernel already supports this, though only with very recent kernels. I'm not 100% sure about mdadm support, but if it isn't there yet, it probably will be soon.

You can convert a RAID10 to a RAID0. You probably have to remove two devices first, so there are just two working devices - no redundancy.

  mdadm --grow /dev/md0 --level=0

Then you can convert the RAID0 to RAID4

  mdadm --grow /dev/md0 --level=4

Then add the good device back in

  mdadm /dev/md0 --add /dev/sdXX

This should all work, though you should certainly test it before you depend on it at all.

NeilBrown

#3 lore raw
In reply to Rogier Wolff

Rogier Wolff put forth on 12/30/2010 3:42 AM:

Quoted 10 lines · Steven Haigh
Quoted 3 lines

Maybe I'm not quite understanding right, however you can easily do RAID6 with 4 drives. That will give you two redundant, effectively give you RAID5 if I drive fails, and save buttloads of messing around...

Steven, My friend has a server where the drives take up to a third of a second to respond. When asking for help, everybody pounced on us:
- NEVER use raid5 for a server doing small-file-io like a mailserver. (always use RAID10).

Quoted 5 lines

So apparently RAID5 (and by extension RAID6) is not an option for some systems.

I'm willing to tolerate the RAID4 situation during the time that it takes me to replace the drive.

Any RAID scheme that uses parity is less than optimal, and up to horrible, for heavy random IO loads. As always, this depends on "how heavy" the load is. For up to a few hundred constant IOPS you can get away with parity RAID schemes. If you need a few thousand or many thousand IOPS, better stay away from parity RAID.

This includes RAID 3 and 4. Both of these are now defunct because using a dedicated disk for storing parity information for an array yields the same or very slightly higher reliability than using a single disk (I don't have the equation in front me to give exact probability of failure). Regardless, if the RAID 3/4 parity disk fails you lose the array.

If your friend's web server isn't going to see a ton of traffic, why does he need anything beyond a 2 way mirror with a spare? Paraniod? Do a 3 way mirror. A mirrored pair of 10k RPM SATA drives should be more than sufficient for most webservers, which typically gain their performance from lots of buffer cache, not from fast disks.

If would help if we knew more about the specific web app he's hosting, its IO patterns, and anticipated load once in production. Unless he's got a super complex (read inefficient) cgi/database back end my recommendation of a pair of mirrored drives, stands. 7.2k would probably be fine, 10k gives a little wiggle room if you underestimate your load target, or the app turns out to be even less efficient that anticipated.

Signature · 1 line
Stan
#4 lore raw

On 30/12/2010 10:39, Stan Hoeppner wrote:
[...]

Quoted 5 lines

Any RAID scheme that uses parity is less than optimal, and up to horrible, for heavy random IO loads. As always, this depends on "how heavy" the load is. For up to a few hundred constant IOPS you can get away with parity RAID schemes. If you need a few thousand or many thousand IOPS, better stay away from parity RAID.

Sorry, I have to disagree with this, in this situation. RAID-6 over 4 discs will be just as fast for reading multiple small files as RAID-10 over 4 discs, and a web server is a read-mostly environment, while at the same time I can't imagine any RAID schema ever giving thousands of IOPS over 4 discs, parity or no.

Cheers,

John.

#5 lore raw

John Robinson put forth on 12/30/2010 5:58 AM:

Quoted 8 lines · Stan Hoeppner

[...]

Quoted 5 lines

Any RAID scheme that uses parity is less than optimal, and up to horrible, for heavy random IO loads. As always, this depends on "how heavy" the load is. For up to a few hundred constant IOPS you can get away with parity RAID schemes. If you need a few thousand or many thousand IOPS, better stay away from parity RAID.

Quoted 5 lines

Sorry, I have to disagree with this, in this situation. RAID-6 over 4 discs will be just as fast for reading multiple small files as RAID-10 over 4 discs, and a web server is a read-mostly environment, while at the same time I can't imagine any RAID schema ever giving thousands of IOPS over 4 discs, parity or no.

That's because you apparently didn't learn about paragraph's in English
class: http://en.wikipedia.org/wiki/Paragraph Do you Brits use paragraphs differently than we do here in the states?

My first paragraph dealt with general performance of parity vs non parity RAID WRT high IO loads. My second paragraph covered the downside of the redundancy methods of RAID 3/4. My third paragraph dealt specifically with Roger's web server.

Note that nothing in my first paragraph mentioned a web server workload.
Also note that nowhere did I mention a count of 4 drive, nor commented
regarding the suitability of any RAID level with 4 drives.

Also note there were two "situations" mentioned by Roger. The first referenced a previous thread which dealt with a high transaction load server similar to a mail server, IIRC. My first paragraph related to that. The second "situation", to which you refer, dealt with Roger's web server.

Signature · 1 line
Stan
#15 lore raw
In reply to Rogier Wolff
Quoted 10 lines · Rogier Wolff

So I was thinking.... He's resigned himself to a configuration where he pays for 4x the disk space and only gets 2x the available space.

So he could run his array in RAID10 mode, however when a drive fails, a fallback to raid5 would be in order. In this case, after the resync a single-drive-failure tolerance is again obtained.

In practise scaling down to raid5 is not easy/possible. RAID4 however should be doable.

Quoted 6 lines

In fact this can almost be implemented entirely in userspace. Just remove the mirror drive from the underlying raid0, and reinitialize as raid4. If you do this correctly the data will still be there....

Although doing this with an active filesystem running on these drives is probably impossible due to "device is in use" error messages....

Paying for 4x the disk space and only getting 2x is about performance. You can't just view disks by the raw space they provide.

Even if the scenario of converting RAID10 to RAID4 was possible you now have no redundancy during the conversion. The chance of a disk failing is greater than just running the RAID10 until you can replace the faulty disk. Not to mention the performance penalty of the resync parity calculation. RAID10 when degraded has minor hit to performance compared to a RAID level with parity.

All you need to do is purchase 5 disks if you want 4 in RAID10. Have the cold spare ready when one fails. This reduces the replacement time as you don't have to wait for a drive to be ordered.

Ryan

#6 lore raw
In reply to Stan Hoeppner
Quoted 15 lines · Stan Hoeppner

John Robinson put forth on 12/30/2010 5:58 AM:

Quoted 8 lines · Stan Hoeppner

[...]

Quoted 5 lines

Any RAID scheme that uses parity is less than optimal, and up to horrible, for heavy random IO loads. As always, this depends on "how heavy" the load is. For up to a few hundred constant IOPS you can get away with parity RAID schemes. If you need a few thousand or many thousand IOPS, better stay away from parity RAID.

Quoted 5 lines

Sorry, I have to disagree with this, in this situation. RAID-6 over 4 discs will be just as fast for reading multiple small files as RAID-10 over 4 discs, and a web server is a read-mostly environment, while at the same time I can't imagine any RAID schema ever giving thousands of IOPS over 4 discs, parity or no.

Quoted 3 lines

That's because you apparently didn't learn about paragraph's in English
class: http://en.wikipedia.org/wiki/Paragraph Do you Brits use paragraphs differently than we do here in the states?

No, but apparently we use apostrophes correctly over here.

Quoted 2 lines

My first paragraph dealt with general performance of parity vs non parity RAID WRT high IO loads.

Yes, and I suppose that I should have pointed out that the OP's friend had been given slightly inappropriate advice, since a web server doesn't do small file I/O like a mailserver. You expanded on a general situation which didn't apply, and the statement you made was wrong, or at least not correct in all circumstances.

Quoted 2 lines

My second paragraph covered the downside
of the redundancy methods of RAID 3/4.

You were wrong again there: if you lose the parity disc in RAID 3/4 you don't lose the array, as the data discs are all still there. It is true that with modern huge (1TB+) drives where the error rate per bit read is still much the same as when drives were tiny (1GB+) that a recovery is much more risky than it used to be due to the dramatically increased chance of a second disc failing, but that is equally true of RAID 5.

Quoted 2 lines

My third paragraph dealt
specifically with Roger's web server.

The third and the fourth; jolly good.

Quoted 3 lines

Note that nothing in my first paragraph mentioned a web server workload.
Also note that nowhere did I mention a count of 4 drive, nor commented
regarding the suitability of any RAID level with 4 drives.

No indeed, but that was the context of the question; why give entirely general advice when a specific usage applies?

Quoted 3 lines

Also note there were two "situations" mentioned by Roger. The first referenced a previous thread which dealt with a high transaction load server similar to a mail server, IIRC.

I see no such reference, apart from noting that "when asking for help, everybody pounced on us: - NEVER use raid5 for a server doing small-file-io like a mailserver. (always use RAID10)" which as I say is in my opinion inappropriate advice, since they're not trying to run a mailserver and won't have heavy random writes.

Quoted 3 lines

My first paragraph related to
that. The second "situation", to which you refer, dealt with Roger's web server.

I had surmised from the original question about using RAID-10, RAID-4 etc that there was a desire to have more storage than a single drive mirrored twice, so I didn't think plain mirroring would suit, but perhaps that wasn't the intention and your solution would work.

Cheers,

John.

#8 lore raw

Re: Why won't mdadm start several RAIDs that appear to be fine?

In reply to Stan Hoeppner

When I rebooted my server yesterday, not all the RAIDs came up. There were no errors in the system log. All devices appear to be working correctly. There is no evidence of hardware errors or data corruption.

To prevent mdadm from failing RAID drives, I removed the RAID entries from /etc/mdadm.conf, and I have a cron script that does things like

mdadm -A --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.

The question is, why did mdadm assemble only 2 drives, when all 4 drives appear to be fine? The same problem occurred for 4 RAIDs, each with similar geometry, and using the same 4 physical drives.

Here is the status of all 4 partitions that should have been assembled into /dev/md5:

[root@l1 ~]# mdadm -E /dev/sda5
/dev/sda5:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
           Name : l1.fu-lab.com:5  (local to host l1.fu-lab.com)

Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)

     Array Size : 2871641088 (1369.31 GiB 1470.28 GB)

Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4088b63f:68d66426:a2abd280:28476493

Update Time : Wed Dec 22 08:27:57 2010
Checksum : 48e371ac - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdi5
/dev/sdi5:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
           Name : l1.fu-lab.com:5  (local to host l1.fu-lab.com)

Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)

     Array Size : 2871641088 (1369.31 GiB 1470.28 GB)

Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : bfc9fe39:c3e40f6a:7418831b:87e08f16

Update Time : Wed Dec 22 08:27:57 2010
Checksum : a4b2c7b7 - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdj5
/dev/sdj5:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
           Name : l1.fu-lab.com:5  (local to host l1.fu-lab.com)

Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)

     Array Size : 2871641088 (1369.31 GiB 1470.28 GB)

Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3e1f7e30:730c70c0:c2770470:8e40ea84

Update Time : Wed Dec 22 08:27:57 2010
Checksum : b46e043d - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdk5
/dev/sdk5:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
           Name : l1.fu-lab.com:5  (local to host l1.fu-lab.com)

Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)

     Array Size : 2871641088 (1369.31 GiB 1470.28 GB)

Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 5acc120a:e7197136:7d7a29c2:971e410d

Update Time : Wed Dec 22 08:27:57 2010
Checksum : de7f9f92 - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing)

I could try assembling the RAID with other command syntaxes (such as by listing all the partitions/devices manually). However, I see no reason why this should be necessary.

Also,

mdadm -V
mdadm - v3.1.2 - 10th March 2010

What is going on?

Thanks!

Jim

#9 lore raw

Re: Why won't mdadm start several RAIDs that appear to be fine?

Quoted 8 lines · On Thu, 30 Dec 2010 16:20:58 -0700 Jim Schatzman…

When I rebooted my server yesterday, not all the RAIDs came up. There were no errors in the system log. All devices appear to be working correctly. There is no evidence of hardware errors or data corruption.

To prevent mdadm from failing RAID drives, I removed the RAID entries from /etc/mdadm.conf, and I have a cron script that does things like

mdadm -A --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.

The question is, why did mdadm assemble only 2 drives, when all 4 drives appear to be fine? The same problem occurred for 4 RAIDs, each with similar geometry, and using the same 4 physical drives.

Add a '--verbose' to the '-A' command. Hopefully it will reveal something interesting.

NeilBrown

Quoted 134 lines

Here is the status of all 4 partitions that should have been assembled into /dev/md5:

[root@l1 ~]# mdadm -E /dev/sda5
/dev/sda5:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
           Name : l1.fu-lab.com:5  (local to host l1.fu-lab.com)

Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)

     Array Size : 2871641088 (1369.31 GiB 1470.28 GB)

Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4088b63f:68d66426:a2abd280:28476493

Update Time : Wed Dec 22 08:27:57 2010
Checksum : 48e371ac - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdi5
/dev/sdi5:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
           Name : l1.fu-lab.com:5  (local to host l1.fu-lab.com)

Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)

     Array Size : 2871641088 (1369.31 GiB 1470.28 GB)

Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : bfc9fe39:c3e40f6a:7418831b:87e08f16

Update Time : Wed Dec 22 08:27:57 2010
Checksum : a4b2c7b7 - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdj5
/dev/sdj5:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
           Name : l1.fu-lab.com:5  (local to host l1.fu-lab.com)

Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)

     Array Size : 2871641088 (1369.31 GiB 1470.28 GB)

Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3e1f7e30:730c70c0:c2770470:8e40ea84

Update Time : Wed Dec 22 08:27:57 2010
Checksum : b46e043d - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)
[root@l1 ~]# mdadm -E /dev/sdk5
/dev/sdk5:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 291655c3:b6c334ff:8dfe69a4:447f777b
           Name : l1.fu-lab.com:5  (local to host l1.fu-lab.com)

Creation Time : Thu Sep 23 13:41:31 2010
Raid Level : raid5
Raid Devices : 4

Avail Dev Size : 957214849 (456.44 GiB 490.09 GB)

     Array Size : 2871641088 (1369.31 GiB 1470.28 GB)

Used Dev Size : 957213696 (456.44 GiB 490.09 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 5acc120a:e7197136:7d7a29c2:971e410d

Update Time : Wed Dec 22 08:27:57 2010
Checksum : de7f9f92 - correct
Events : 339

Layout : left-symmetric
Chunk Size : 512K

Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing)

I could try assembling the RAID with other command syntaxes (such as by listing all the partitions/devices manually). However, I see no reason why this should be necessary.

Also,

mdadm -V
mdadm - v3.1.2 - 10th March 2010

What is going on?

Thanks!

Jim

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw

Re: Why won't mdadm start several RAIDs that appear to be fine? Info from "mdadm -A --verbose"

After explicitly stopping the RAID (mdadm -S /dev/md5), and executing the mdadm -A --verbose command as suggested, more info is forthcoming. Mdadm appears to think that /dev/sdi5 and /dev/sdj5 are "busy". Since only /dev/sda5 and /dev/sdk5 can be added to the RAID, we are hosed.

I am puzzled as to why it thinks /dev/sdi5 and /dev/sdj5 are "busy". Fdisk reports normal partition data, dmesg and the system log report no problems, and I have no trouble copying data from /dev/sdi5 and /dev/sdj5 with dd. The system log does contain messages like

"kernel: [ 47.946357] dracut: Scanning devices md0 md1 md2 md3 sdi sdi1 sdi2 sdj sdj1 sdj2 sdk sdk1 sdk2 sdl for LVM volume groups"

but there appears to be no essential difference in the logging messages for sdi,j versus sda,k. Also, the physical devices (a,i,j,k) are the same brand and model number disk drive. The system log reports

"3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)"

for all of them. Also, "fuser -m /dev/sdi5" and the same for sdj5 reports no users of the filesystems.

Stopping /dev/md5,6,7,8 and rerunning mdadm -A command, shows that mdadm also thinks that /dev/sdi6,7,8 and /dev/sdj6,7,8 are busy as well, even though they are not being used in any other RAID.

What does this mean?

I am grasping at straws here. It seems that mdadm thinks that two of the devices are "busy", even though Linux apparently disagrees, and no I/O errors are being reported. I am mystified.

After googling, I did find one suggestion - remove dmraid from the system. I will try that and report the result. I am not sure why dmraid is installed - I never explicitly installed it. I see that I have

dmraid-events-1.0.0.rc16-12.fc13.i686
dmraid-1.0.0.rc16-12.fc13.i686

Thanks!

Jim

1st try
-----------------

mdadm -A --verbose --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
mdadm: looking for devices for /dev/md5
mdadm: no recogniseable superblock on /dev/dm-6
mdadm: /dev/dm-6 has wrong uuid.
mdadm: no recogniseable superblock on /dev/dm-5
mdadm: /dev/dm-5 has wrong uuid.
mdadm: cannot open device /dev/dm-4: Device or resource busy
mdadm: /dev/dm-4 has wrong uuid.
mdadm: cannot open device /dev/dm-3: Device or resource busy
mdadm: /dev/dm-3 has wrong uuid.
mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: /dev/dm-2 has wrong uuid.
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: /dev/dm-1 has wrong uuid.
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: cannot open device /dev/md2: Device or resource busy
mdadm: /dev/md2 has wrong uuid.
mdadm: cannot open device /dev/md3: Device or resource busy
mdadm: /dev/md3 has wrong uuid.
mdadm: cannot open device /dev/md0: Device or resource busy
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: /dev/dm-0 has wrong uuid.
mdadm: cannot open device /dev/sdm7: Device or resource busy
mdadm: /dev/sdm7 has wrong uuid.
mdadm: cannot open device /dev/sdm6: Device or resource busy
mdadm: /dev/sdm6 has wrong uuid.
mdadm: cannot open device /dev/sdm5: Device or resource busy
mdadm: /dev/sdm5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdm1
mdadm: /dev/sdm1 has wrong uuid.
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: /dev/sdm has wrong uuid.
mdadm: cannot open device /dev/sdp7: Device or resource busy
mdadm: /dev/sdp7 has wrong uuid.
mdadm: cannot open device /dev/sdp6: Device or resource busy
mdadm: /dev/sdp6 has wrong uuid.
mdadm: cannot open device /dev/sdp5: Device or resource busy
mdadm: /dev/sdp5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdp1
mdadm: /dev/sdp1 has wrong uuid.
mdadm: cannot open device /dev/sdp: Device or resource busy
mdadm: /dev/sdp has wrong uuid.
mdadm: cannot open device /dev/sdo7: Device or resource busy
mdadm: /dev/sdo7 has wrong uuid.
mdadm: cannot open device /dev/sdo6: Device or resource busy
mdadm: /dev/sdo6 has wrong uuid.
mdadm: cannot open device /dev/sdo5: Device or resource busy
mdadm: /dev/sdo5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdo1
mdadm: /dev/sdo1 has wrong uuid.
mdadm: cannot open device /dev/sdo: Device or resource busy
mdadm: /dev/sdo has wrong uuid.
mdadm: cannot open device /dev/sdn7: Device or resource busy
mdadm: /dev/sdn7 has wrong uuid.
mdadm: cannot open device /dev/sdn6: Device or resource busy
mdadm: /dev/sdn6 has wrong uuid.
mdadm: cannot open device /dev/sdn5: Device or resource busy
mdadm: /dev/sdn5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdn1
mdadm: /dev/sdn1 has wrong uuid.
mdadm: cannot open device /dev/sdn: Device or resource busy
mdadm: /dev/sdn has wrong uuid.
mdadm: cannot open device /dev/sdi8: Device or resource busy
mdadm: /dev/sdi8 has wrong uuid.
mdadm: cannot open device /dev/sdi7: Device or resource busy
mdadm: /dev/sdi7 has wrong uuid.
mdadm: cannot open device /dev/sdi6: Device or resource busy
mdadm: /dev/sdi6 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: cannot open device /dev/sdi2: Device or resource busy
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: /dev/sdl has wrong uuid.
mdadm: cannot open device /dev/sdk8: Device or resource busy
mdadm: /dev/sdk8 has wrong uuid.
mdadm: cannot open device /dev/sdk7: Device or resource busy
mdadm: /dev/sdk7 has wrong uuid.
mdadm: cannot open device /dev/sdk6: Device or resource busy
mdadm: /dev/sdk6 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk2
mdadm: /dev/sdk2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk1
mdadm: /dev/sdk1 has wrong uuid.
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: /dev/sdk has wrong uuid.
mdadm: cannot open device /dev/sdj8: Device or resource busy
mdadm: /dev/sdj8 has wrong uuid.
mdadm: cannot open device /dev/sdj7: Device or resource busy
mdadm: /dev/sdj7 has wrong uuid.
mdadm: cannot open device /dev/sdj6: Device or resource busy
mdadm: /dev/sdj6 has wrong uuid.
mdadm: cannot open device /dev/sdj5: Device or resource busy
mdadm: /dev/sdj5 has wrong uuid.
mdadm: cannot open device /dev/sdj2: Device or resource busy
mdadm: /dev/sdj2 has wrong uuid.
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: /dev/sdj1 has wrong uuid.
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: /dev/sdj has wrong uuid.
mdadm: cannot open device /dev/sdh3: Device or resource busy
mdadm: /dev/sdh3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh2
mdadm: /dev/sdh2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh1
mdadm: /dev/sdh1 has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: /dev/sdf3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdf2
mdadm: /dev/sdf2 has wrong uuid.
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: /dev/sdf1 has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sdg3: Device or resource busy
mdadm: /dev/sdg3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg2
mdadm: /dev/sdg2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg1
mdadm: /dev/sdg1 has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdb7: Device or resource busy
mdadm: /dev/sdb7 has wrong uuid.
mdadm: cannot open device /dev/sdb6: Device or resource busy
mdadm: /dev/sdb6 has wrong uuid.
mdadm: cannot open device /dev/sdb5: Device or resource busy
mdadm: /dev/sdb5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdb1
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sde7: Device or resource busy
mdadm: /dev/sde7 has wrong uuid.
mdadm: cannot open device /dev/sde6: Device or resource busy
mdadm: /dev/sde6 has wrong uuid.
mdadm: cannot open device /dev/sde5: Device or resource busy
mdadm: /dev/sde5 has wrong uuid.
mdadm: no RAID superblock on /dev/sde1
mdadm: /dev/sde1 has wrong uuid.
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd7: Device or resource busy
mdadm: /dev/sdd7 has wrong uuid.
mdadm: cannot open device /dev/sdd6: Device or resource busy
mdadm: /dev/sdd6 has wrong uuid.
mdadm: cannot open device /dev/sdd5: Device or resource busy
mdadm: /dev/sdd5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdd1
mdadm: /dev/sdd1 has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc7: Device or resource busy
mdadm: /dev/sdc7 has wrong uuid.
mdadm: cannot open device /dev/sdc6: Device or resource busy
mdadm: /dev/sdc6 has wrong uuid.
mdadm: cannot open device /dev/sdc5: Device or resource busy
mdadm: /dev/sdc5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdc1
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: cannot open device /dev/sda8: Device or resource busy
mdadm: /dev/sda8 has wrong uuid.
mdadm: cannot open device /dev/sda7: Device or resource busy
mdadm: /dev/sda7 has wrong uuid.
mdadm: cannot open device /dev/sda6: Device or resource busy
mdadm: /dev/sda6 has wrong uuid.
mdadm: no RAID superblock on /dev/sda2
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/sdk5 is identified as a member of /dev/md5, slot 3.
mdadm: /dev/sda5 is identified as a member of /dev/md5, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md5
mdadm: no uptodate device for slot 2 of /dev/md5
mdadm: added /dev/sdk5 to /dev/md5 as 3
mdadm: added /dev/sda5 to /dev/md5 as 0
mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.

Stopping /dev/md5,6,7,8 first
------------------------------------------

mdadm -A --verbose --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
mdadm: looking for devices for /dev/md5
mdadm: no recogniseable superblock on /dev/dm-6
mdadm: /dev/dm-6 has wrong uuid.
mdadm: no recogniseable superblock on /dev/dm-5
mdadm: /dev/dm-5 has wrong uuid.
mdadm: cannot open device /dev/dm-4: Device or resource busy
mdadm: /dev/dm-4 has wrong uuid.
mdadm: cannot open device /dev/dm-3: Device or resource busy
mdadm: /dev/dm-3 has wrong uuid.
mdadm: cannot open device /dev/dm-2: Device or resource busy
mdadm: /dev/dm-2 has wrong uuid.
mdadm: cannot open device /dev/dm-1: Device or resource busy
mdadm: /dev/dm-1 has wrong uuid.
mdadm: cannot open device /dev/md1: Device or resource busy
mdadm: /dev/md1 has wrong uuid.
mdadm: cannot open device /dev/md2: Device or resource busy
mdadm: /dev/md2 has wrong uuid.
mdadm: cannot open device /dev/md3: Device or resource busy
mdadm: /dev/md3 has wrong uuid.
mdadm: cannot open device /dev/md0: Device or resource busy
mdadm: /dev/md0 has wrong uuid.
mdadm: cannot open device /dev/dm-0: Device or resource busy
mdadm: /dev/dm-0 has wrong uuid.
mdadm: cannot open device /dev/sdm7: Device or resource busy
mdadm: /dev/sdm7 has wrong uuid.
mdadm: cannot open device /dev/sdm6: Device or resource busy
mdadm: /dev/sdm6 has wrong uuid.
mdadm: cannot open device /dev/sdm5: Device or resource busy
mdadm: /dev/sdm5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdm1
mdadm: /dev/sdm1 has wrong uuid.
mdadm: cannot open device /dev/sdm: Device or resource busy
mdadm: /dev/sdm has wrong uuid.
mdadm: cannot open device /dev/sdp7: Device or resource busy
mdadm: /dev/sdp7 has wrong uuid.
mdadm: cannot open device /dev/sdp6: Device or resource busy
mdadm: /dev/sdp6 has wrong uuid.
mdadm: cannot open device /dev/sdp5: Device or resource busy
mdadm: /dev/sdp5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdp1
mdadm: /dev/sdp1 has wrong uuid.
mdadm: cannot open device /dev/sdp: Device or resource busy
mdadm: /dev/sdp has wrong uuid.
mdadm: cannot open device /dev/sdo7: Device or resource busy
mdadm: /dev/sdo7 has wrong uuid.
mdadm: cannot open device /dev/sdo6: Device or resource busy
mdadm: /dev/sdo6 has wrong uuid.
mdadm: cannot open device /dev/sdo5: Device or resource busy
mdadm: /dev/sdo5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdo1
mdadm: /dev/sdo1 has wrong uuid.
mdadm: cannot open device /dev/sdo: Device or resource busy
mdadm: /dev/sdo has wrong uuid.
mdadm: cannot open device /dev/sdn7: Device or resource busy
mdadm: /dev/sdn7 has wrong uuid.
mdadm: cannot open device /dev/sdn6: Device or resource busy
mdadm: /dev/sdn6 has wrong uuid.
mdadm: cannot open device /dev/sdn5: Device or resource busy
mdadm: /dev/sdn5 has wrong uuid.
mdadm: no recogniseable superblock on /dev/sdn1
mdadm: /dev/sdn1 has wrong uuid.
mdadm: cannot open device /dev/sdn: Device or resource busy
mdadm: /dev/sdn has wrong uuid.
mdadm: cannot open device /dev/sdi8: Device or resource busy
mdadm: /dev/sdi8 has wrong uuid.
mdadm: cannot open device /dev/sdi7: Device or resource busy
mdadm: /dev/sdi7 has wrong uuid.
mdadm: cannot open device /dev/sdi6: Device or resource busy
mdadm: /dev/sdi6 has wrong uuid.
mdadm: cannot open device /dev/sdi5: Device or resource busy
mdadm: /dev/sdi5 has wrong uuid.
mdadm: cannot open device /dev/sdi2: Device or resource busy
mdadm: /dev/sdi2 has wrong uuid.
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: /dev/sdi1 has wrong uuid.
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: /dev/sdi has wrong uuid.
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: /dev/sdl has wrong uuid.
mdadm: /dev/sdk8 has wrong uuid.
mdadm: /dev/sdk7 has wrong uuid.
mdadm: /dev/sdk6 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk2
mdadm: /dev/sdk2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk1
mdadm: /dev/sdk1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdk
mdadm: /dev/sdk has wrong uuid.
mdadm: cannot open device /dev/sdj8: Device or resource busy
mdadm: /dev/sdj8 has wrong uuid.
mdadm: cannot open device /dev/sdj7: Device or resource busy
mdadm: /dev/sdj7 has wrong uuid.
mdadm: cannot open device /dev/sdj6: Device or resource busy
mdadm: /dev/sdj6 has wrong uuid.
mdadm: cannot open device /dev/sdj5: Device or resource busy
mdadm: /dev/sdj5 has wrong uuid.
mdadm: cannot open device /dev/sdj2: Device or resource busy
mdadm: /dev/sdj2 has wrong uuid.
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: /dev/sdj1 has wrong uuid.
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: /dev/sdj has wrong uuid.
mdadm: cannot open device /dev/sdh3: Device or resource busy
mdadm: /dev/sdh3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh2
mdadm: /dev/sdh2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh1
mdadm: /dev/sdh1 has wrong uuid.
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: /dev/sdh has wrong uuid.
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: /dev/sdf3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdf2
mdadm: /dev/sdf2 has wrong uuid.
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: /dev/sdf1 has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: cannot open device /dev/sdg3: Device or resource busy
mdadm: /dev/sdg3 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg2
mdadm: /dev/sdg2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg1
mdadm: /dev/sdg1 has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdb7: Device or resource busy
mdadm: /dev/sdb7 has wrong uuid.
mdadm: cannot open device /dev/sdb6: Device or resource busy
mdadm: /dev/sdb6 has wrong uuid.
mdadm: cannot open device /dev/sdb5: Device or resource busy
mdadm: /dev/sdb5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdb1
mdadm: /dev/sdb1 has wrong uuid.
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sde7: Device or resource busy
mdadm: /dev/sde7 has wrong uuid.
mdadm: cannot open device /dev/sde6: Device or resource busy
mdadm: /dev/sde6 has wrong uuid.
mdadm: cannot open device /dev/sde5: Device or resource busy
mdadm: /dev/sde5 has wrong uuid.
mdadm: no RAID superblock on /dev/sde1
mdadm: /dev/sde1 has wrong uuid.
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd7: Device or resource busy
mdadm: /dev/sdd7 has wrong uuid.
mdadm: cannot open device /dev/sdd6: Device or resource busy
mdadm: /dev/sdd6 has wrong uuid.
mdadm: cannot open device /dev/sdd5: Device or resource busy
mdadm: /dev/sdd5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdd1
mdadm: /dev/sdd1 has wrong uuid.
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has wrong uuid.
mdadm: cannot open device /dev/sdc7: Device or resource busy
mdadm: /dev/sdc7 has wrong uuid.
mdadm: cannot open device /dev/sdc6: Device or resource busy
mdadm: /dev/sdc6 has wrong uuid.
mdadm: cannot open device /dev/sdc5: Device or resource busy
mdadm: /dev/sdc5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdc1
mdadm: /dev/sdc1 has wrong uuid.
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: /dev/sdc has wrong uuid.
mdadm: /dev/sda8 has wrong uuid.
mdadm: /dev/sda7 has wrong uuid.
mdadm: /dev/sda6 has wrong uuid.
mdadm: no RAID superblock on /dev/sda2
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: /dev/sda1 has wrong uuid.
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/sdk5 is identified as a member of /dev/md5, slot 3.
mdadm: /dev/sda5 is identified as a member of /dev/md5, slot 0.
mdadm: no uptodate device for slot 1 of /dev/md5
mdadm: no uptodate device for slot 2 of /dev/md5
mdadm: added /dev/sdk5 to /dev/md5 as 3
mdadm: added /dev/sda5 to /dev/md5 as 0
mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.

Quoted 17 lines · Neil Brown
Quoted 8 lines · On Thu, 30 Dec 2010 16:20:58 -0700 Jim Schatzman…

When I rebooted my server yesterday, not all the RAIDs came up. There were no errors in the system log. All devices appear to be working correctly. There is no evidence of hardware errors or data corruption.

To prevent mdadm from failing RAID drives, I removed the RAID entries from /etc/mdadm.conf, and I have a cron script that does things like

mdadm -A --no-degraded /dev/md5 --uuid 291655c3:b6c334ff:8dfe69a4:447f777b
mdadm: /dev/md5 assembled from 2 drives (out of 4), but not started.

The question is, why did mdadm assemble only 2 drives, when all 4 drives appear to be fine? The same problem occurred for 4 RAIDs, each with similar geometry, and using the same 4 physical drives.

Add a '--verbose' to the '-A' command. Hopefully it will reveal something interesting.

NeilBrown

#11 lore raw

Re: Why won't mdadm start several RAIDs that appear to be fine? SOLVED!

In reply to Neil Brown

All-

Using yum/rpm to remove dmraid from the system and rebooting fixed the problem.

Why is dmraid doing anything at all when my motherboard doesn't support FakeRAID? It seems that it is arbitrarily picking some set of drives to tie up, effectively disabling mdadm for some arrays. Nice.

Jim

#7 lore raw
In reply to John Robinson

John Robinson put forth on 12/30/2010 12:10 PM:

Quoted 4 lines

I had surmised from the original question about using RAID-10, RAID-4 etc that there was a desire to have more storage than a single drive mirrored twice, so I didn't think plain mirroring would suit, but perhaps that wasn't the intention and your solution would work.

Sorry I was a bit prickly in my reply John. For some reason I became defensive, and shouldn't have. Chalk it up to mood I guess.

It's entirely possible that I misunderstood Roger's requirements. I believe he was talking about two different systems, one a transaction type server in his first thread, the other just a web sever in this thread. That's why I recommended the possibility of simple RAID 1 for the web server.

It's difficult for me to imagine a web server scenario that would need anywhere close to 1TB of disk, or one that would need more IOPS than a single disk could provide, or more fault tolerance than mirroring. The assumption today being that one satisfies web capacity needs with many cheap nodes instead of one, or few, big ones. I concede anything is possible, and there are myriad requirements out there. I've just never seen/heard of a web server req for anything more than simple disk mirroring.

For instance, I've been using the following for a web node with good success. It's a "low power" node from both an all out performance and heat dissipation perspective but can handle more than sufficient numbers of simultaneous requests (it is noisy though, as all 1U units are). Current cost of the components is less than $360 USD for a 1U 14" deep single core 2.8GHz 45w AMD server, 4GB RAM, onboard single GigE, and 2 x mirrored Seagate 160GB 7.2k 2.5" SATA II drives, and a single 260w PSU.
These boxen don't have hot swap drive cages. Using a box with hot swap
would increase total price by 35% to almost $500 per node. Drive failures are rare enough here that it's not a burden to de rack the server and replace the dive, as this is a cluster web node. For most other server applications I use hot swap chassis (and redundant PSUs). I run Debian Lenny on these w/lighttpd, etc.

These shallow boxen allow dog ear mounting without making me nervous, so I save about $25-$40 per unit on slide rails. I published the NewEgg wish list of the parts for this build. It should be available in a day or so after their review process. If anyone is interested, search the public wish lists for "web-node". For $70 more you can drop in a 3 core 2.5 GHz AMD and stay at 45w, if a single core 2.8 isn't enough for your needs. You can run up to 65w CPUs in this setup, including the latest Phenom II 45nm quad cores. I try to stick to 45w myself for 1U boxen. Note these are decidedly desktop quality parts, not server, thus the < $400 cost. I find them more than sufficiently reliable for web nodes.

Note that the SuperMicro chassis doesn't natively support 2.5" drives, only two 3.5" drives. I remedy this easily with a template I made, my DeWalt cordless drill, and a countersink bit. Takes less than 5 minutes to drill the 8 new mounting holes. I much prefer the decreased obstruction and increased airflow afforded by 2.5" drives so it's more than worth the trouble. For those averse to drilling on their chassis, NewEgg has a ~$9 adapter plate which works for this application. You'd need two of them for two 2.5" drives.
http://www.newegg.com/Product/Product.aspx?Item=N82E16811993005
(Simply toss the dongle that comes with it as it's not needed)

Apologies for getting a little OT here. A quick glance at the RHS of my email address will explain. :)

Signature · 1 line
Stan
#12 lore raw
In reply to Rogier Wolff
Quoted 10 lines

-----Original Message-----
From: [email protected] [mailto:linux-raid- [email protected]] On Behalf Of Rogier Wolff
Sent: Thursday, December 30, 2010 3:43 AM
To: Steven Haigh
Cc: Rogier Wolff; [email protected]
Subject: Re: New raid level suggestion.

Quoted 4 lines
Quoted 3 lines · Steven Haigh

Maybe I'm not quite understanding right, however you can easily do RAID6 with 4 drives. That will give you two redundant, effectively give you RAID5 if I drive fails, and save buttloads of messing around...

Theree's been quite a bit of back and forth in this thread. I think
it would be best if you could more narrowly define your application. Exactly what is this app doing? Is it, as has been suggested, a web server? How many transactions / second is it servicing at peak? How large are the files? Is there some unusual .cgi script which causes huge amounts of disk thrashing? You might post the results of iostat.

Quoted 2 lines

Steven, My friend has a server where the drives take up to a third of a second to respond.

Respond to what? I have a .cgi script that takes up to 30 seconds
to respond, but it's not because of any lack of array responsiveness. It's performing all sorts of investigations and calculations. 1/3of a second may or may not be a terrible delay depending on what is going on, and the delay may not be as a result of disk I/O.

Quoted 3 lines

When asking for help, everybody pounced on us:
- NEVER use raid5 for a server doing small-file-io like a mailserver. (always use RAID10).

Even a mailserver may not need anything radical in terms of disk
performance, depending on the number of users. Again, you haven't quantified the number of users the server is tending.

Quoted 5 lines

So apparently RAID5 (and by extension RAID6) is not an option for some systems.

I'm willing to tolerate the RAID4 situation during the time that it takes me to replace the drive.

A hot spare can certainly mitigate any windshield time, but before
anyone can really determine that RAID5 or RAID5 is not sufficient, one must specify the actual service parameters.

#13 lore raw
Quoted 10 lines · Leslie Rhorer
Quoted 10 lines

-----Original Message-----
From: [email protected] [mailto:linux-raid- [email protected]] On Behalf Of Rogier Wolff
Sent: Thursday, December 30, 2010 3:43 AM
To: Steven Haigh
Cc: Rogier Wolff; [email protected]
Subject: Re: New raid level suggestion.

Quoted 8 lines
Quoted 4 lines · Steven Haigh
Quoted 3 lines

Maybe I'm not quite understanding right, however you can easily do RAID6 with 4 drives. That will give you two redundant, effectively give you RAID5 if I drive fails, and save buttloads of messing around...

Theree's been quite a bit of back and forth in this thread. I think
it would be best if you could more narrowly define your application. Exactly what is this app doing? Is it, as has been suggested, a web server?

App? Yes, its' a web/mail server for a few small domains.

Quoted 1 line

How many transactions / second is it servicing at peak?

Not all that many. It handles a few thousand Emails a day.

Quoted 2 lines

How large are the
files?

Emails? A few kb. Maybe 10kb on average.

Quoted 2 lines

Is there some unusual .cgi script which causes huge amounts of disk thrashing? You might post the results of iostat.

No.

Quoted 4 lines
Quoted 2 lines

Steven, My friend has a server where the drives take up to a third of a second to respond.

Respond to what?

Read what I wrote. The DISK DRIVE takes up to a third of a second to respond to an IO request.

iostat reports this when started with the -x option.

When the DISK takes a third of a second to respond, the load can skyrocket to say 20. You'll then find that 18 of those are in the queue waiting for one of the disks, giving an average waiting time for the result of the io request (thus queue + service time of the disk) of around 6 seconds. That's when the server feels laggy.... You type a command, some of the data needs to come from that drive, and then it takes up to 6 seconds for the results to come back.

I have been running iostat -x on several different machines and webservers and none of the disks happen to take more than 20ms to respond to an IO request.

There is something wrong with that machine. Although some have suggested that the RAID config is not ideal for high throughput, it should work reasonably for this low-performance server. Although some have suggested that these disks are not ideal for this load, they should be able to respond to IO requests at a higher rate than 3 per second.

I do not think the drives are bad. I expect to be able to test the drives at high throughput and high IO-rate once the server is replaced by a new server.

I suspect that something is wrong with the machine. Something like interrupts for "IO DONE" for the sata controllers not getting delivered immediately. However everybody so far has been shouting:
bad disks! bad raid config!

Quoted 5 lines

I have a .cgi script that takes up to 30 seconds to respond, but it's not because of any lack of array responsiveness. It's performing all sorts of investigations and calculations. 1/3of a second may or may not be a terrible delay depending on what is going on, and the delay may not be as a result of disk I/O.

We measure delays of up to tens of seconds for things that should take less than a tenth of a second, and we've narrowed it down to the disks being slow to respond.

Quoted 7 lines
Quoted 3 lines

When asking for help, everybody pounced on us:
- NEVER use raid5 for a server doing small-file-io like a mailserver. (always use RAID10).

Even a mailserver may not need anything radical in terms of disk
performance, depending on the number of users. Again, you haven't quantified the number of users the server is tending.

On a different server, there are 7 users. 18k Emails/day. This server I think about 10 times more users, 1k Emails/day.

So why do you want to know the number of users? The number of Emails is relevant.

Quoted 9 lines
Quoted 5 lines

So apparently RAID5 (and by extension RAID6) is not an option for some systems.

I'm willing to tolerate the RAID4 situation during the time that it takes me to replace the drive.

A hot spare can certainly mitigate any windshield time, but before anyone can really determine that RAID5 or RAID5 is not sufficient, one must specify the actual service parameters.

Hmm. This seems to be in response to my plan of a new raid config. On a different mailing list we've had tons of useless discussions about how wrong that machine was configured after I asked if someone knew how to find out why the disks were taking so long to respond.

This seems to be moving in the same direction. In this thread I'm NOT fishing for help with that server. (Although if you know of a way how to figure out why those disks (seem to) respond so slowly you're welcome).

What this is about is:

People suggest that RAID 5 is not appropriate for a medium-to-high traffic mailserver, so you'd run raid 10.

However, running RAID10 has the disadvantage that when one disk fails, you're open to dataloss with a large window if it takes you up to a week to replace the failed drive. (which is typical in my and my friends case, and it's acceptable for me and my friends application)

So what I suggested is that once you have a setup where you're happy with only half the total disk space, you can run RAID10 for speed, and convert to RAID4 after one disk fails. This in practice would cover about 98% of the time between the disk failing and the replacement disk arriving, reducing the chances of "dataloss" by about 50 fold.

You'll have less performance with the downed drive, but it's much better to have good performance 99% of the time (when you have 4/4 disks) and bad performance during 1% of the time (when you have only 3/4 disks available) than having bad performance all the time (because of the bad write performance of raid5 and raid6.)

Roger.

Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ