Skip to contentSkip to conversation

very degraded RAID5, or increasing capacity by adding discs

DISCUSSION Janek Kozicki 9 participants 17 messages Oct 8, 2007 – Oct 23, 2007 MD RAID

lore
Share & tools
lore
#0 lore raw

Hello,

Recently I started to use mdadm and I'm very impressed by its capabilities.

I have raid0 (250+250 GB) on my workstation. And I want to have raid5 (4*500 = 1500 GB) on my backup machine.

The backup machine currently doesn't have raid, just a single 500 GB drive. I plan to buy more HDDs to have a bigger space for my backups but since I cannot afford all HDDs at once I face a problem of "expanding" an array. I'm able to add one 500 GB drive every few months until I have all 4 drives.

But I cannot make a backup of a backup... so reformatting/copying all data each time when I add new disc to the array is not possible for me.

Is it possible anyhow to create a "very degraded" raid array - a one that consists of 4 drives, but has only TWO ?

This would involve some very tricky *hole* management on the block device... A one that places holes in stripes on the block device, until more discs are added to fill the holes. When the holes are filled, the block device grows bigger, and with lvm I just increase the filesystem size. This is perhaps coupled with some "unstripping" that moves/reorganizes blocks around to fill/defragment the holes.

is it just a pipe dream?

best regards

PS: yes it's simple to make a degraded array of 3 drives, but I cannot afford two discs at once...

Signature · 1 line
Janek Kozicki                                                         |
#1 lore raw
Quoted 27 lines · Janek Kozicki

Hello,

Recently I started to use mdadm and I'm very impressed by its capabilities.

I have raid0 (250+250 GB) on my workstation. And I want to have raid5 (4*500 = 1500 GB) on my backup machine.

The backup machine currently doesn't have raid, just a single 500 GB drive. I plan to buy more HDDs to have a bigger space for my backups but since I cannot afford all HDDs at once I face a problem of "expanding" an array. I'm able to add one 500 GB drive every few months until I have all 4 drives.

But I cannot make a backup of a backup... so reformatting/copying all data each time when I add new disc to the array is not possible for me.

Is it possible anyhow to create a "very degraded" raid array - a one that consists of 4 drives, but has only TWO ?

This would involve some very tricky *hole* management on the block device... A one that places holes in stripes on the block device, until more discs are added to fill the holes. When the holes are filled, the block device grows bigger, and with lvm I just increase the filesystem size. This is perhaps coupled with some "unstripping" that moves/reorganizes blocks around to fill/defragment the holes.

Quoted 15 lines

is it just a pipe dream?

best regards

PS: yes it's simple to make a degraded array of 3 drives, but I cannot afford two discs at once...

Signature · 5 lines
Janek Kozicki                                                         |
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

With raid1 you can create a degraded array with 1 disk- I have done this, I have always wondered if mdadm will let you make a degraded raid 5 array with 2 disks (you'd specify 3 and only give 2) - you can always expand later.

Justin.

#2 lore raw
In reply to Janek Kozicki
Quoted 2 lines · Janek Kozicki

Is it possible anyhow to create a "very degraded" raid array - a one that consists of 4 drives, but has only TWO ?

No, but you can make a degraded 3 drive array, containing 2 drives and then add the next drive to complete it.

The array can then be grown (man mdadm, GROW section), to add the fourth.

Regards,

Richard

#3 lore raw

} -----Original Message-----
} From: [email protected] [mailto:linux-raid-
} [email protected]] On Behalf Of Richard Scobie
} Sent: Monday, October 08, 2007 3:27 PM
} To: [email protected]
} Subject: Re: very degraded RAID5, or increasing capacity by adding discs }
} Janek Kozicki wrote:
}
} > Is it possible anyhow to create a "very degraded" raid array - a one } > that consists of 4 drives, but has only TWO ?
}
} No, but you can make a degraded 3 drive array, containing 2 drives and } then add the next drive to complete it.
}
} The array can then be grown (man mdadm, GROW section), to add the fourth. }
} Regards,
}
} Richard

I think someone once said you could create a 2 disk degraded RAID5 array with just 1 disk. Then add one later. Then expand as needed. Someone should test this.

#4 lore raw
In reply to Richard Scobie

Richard Scobie said: (by the date of Tue, 09 Oct 2007 08:26:35 +1300)

Quoted 4 lines

No, but you can make a degraded 3 drive array, containing 2 drives and then add the next drive to complete it.

The array can then be grown (man mdadm, GROW section), to add the fourth.

Oh, good. Thanks, I must've been blind that I missed this. This completely solves my problem.

Signature · 1 line
Janek Kozicki                                                         |
#5 lore raw

Janek Kozicki said: (by the date of Tue, 9 Oct 2007 00:25:50 +0200)

Quoted 9 lines

Richard Scobie said: (by the date of Tue, 09 Oct 2007 08:26:35 +1300)

Quoted 4 lines

No, but you can make a degraded 3 drive array, containing 2 drives and then add the next drive to complete it.

The array can then be grown (man mdadm, GROW section), to add the fourth.

Oh, good. Thanks, I must've been blind that I missed this. This completely solves my problem.

Uh, actually not :)

My 1st 500 GB drive is full now. When I buy a 2nd one I want to create a 3-disc degraded array using just 2 discs, one of which contains unbackupable data.

steps:
1. create degraded two-disc RAID5 on 1 new disc
2. copy data from old disc to new one
3. rebuild the array with old and new discs (now I have 500 GB on 2 discs)
4. GROW this array to a degraded 3 discs RAID5 (so I have 1000 GB on 2 discs)
...
5. when I buy 3rd drive I either grow the array, or just rebuild and wait with growing until I buy a 4th drive.

Problems at step 4.: 'man mdadm' doesn't tell if it's possible to grow an array to a degraded array (non existant disc). Is it possible?

PS: the fact, that degraded array will be unsafe for the data is an intented motivating factor for buying next drive ;)

Signature · 1 line
Janek Kozicki                                                         |
#11 lore raw
In reply to Janek Kozicki
Quoted 7 lines · Janek Kozicki

Hello,

Recently I started to use mdadm and I'm very impressed by its capabilities.

I have raid0 (250+250 GB) on my workstation. And I want to have raid5 (4*500 = 1500 GB) on my backup machine.

Hmm. Are you sure you need that much space on the backup, to start with? Maybe better backup strategy will help to avoid hardware costs? Such as using rsync for backups as discussed on this mailinglist about a month back (rsync is able to keep many ready to use copies of your filesystems but only store files that actually changed since the last backup, thus requiring much less space than many full backups).

Quoted 12 lines

The backup machine currently doesn't have raid, just a single 500 GB drive. I plan to buy more HDDs to have a bigger space for my backups but since I cannot afford all HDDs at once I face a problem of "expanding" an array. I'm able to add one 500 GB drive every few months until I have all 4 drives.

But I cannot make a backup of a backup... so reformatting/copying all data each time when I add new disc to the array is not possible for me.

Is it possible anyhow to create a "very degraded" raid array - a one that consists of 4 drives, but has only TWO ?

Quoted 6 lines

This would involve some very tricky *hole* management on the block device... A one that places holes in stripes on the block device, until more discs are added to fill the holes. When the holes are filled, the block device grows bigger, and with lvm I just increase the filesystem size. This is perhaps coupled with some "unstripping" that moves/reorganizes blocks around to fill/defragment the holes.

It's definitely not possible with raid5. Only option is to create a raid5 array consisting of less drives than it should contain at the end, and reshape it when you get more drives, as others noted in this thread. But do note the following points:

o degraded raid5 isn't really Raid - i.e, it's not any better than a raid0 array, that is, any disk fails => the whole array fails. So instead of creating a degraded raid5 array initially, create smaller one instead, but not degraded, and reshape it when necessary.

o reshaping takes time, and for this volume, reshape will take many hours, maybe days, to complete.

o During this reshape time, errors may be fatal to the whole array - while mdadm do have a sense of "critical section", but the whole procedure isn't as much tested as the rest of raid code, I for one will not rely on it, at least for now. For example, a power failure at an "unexpected" moment, or some plain-stupid error in reshape code so that the whole array goes "boom" etc...

o A filesystem on the array has to be resized separately after re{siz,shap}ing the array. And filesystems are different at this point, too - there are various limitations. For example, it's problematic to grow ext[23]fs by large amounts, because when it gets initially created, mke2fs calculates sizes of certain internal data structures based on the device size, and those structures can't be grown significantly, only recreating the filesystem will do the trick.

Quoted 1 line

is it just a pipe dream?

I'd say it is... ;)

/mjt

#6 lore raw
In reply to Janek Kozicki

} -----Original Message-----
} From: [email protected] [mailto:linux-raid-
} [email protected]] On Behalf Of Janek Kozicki
} Sent: Monday, October 08, 2007 6:47 PM
} To: [email protected]
} Subject: Re: very degraded RAID5, or increasing capacity by adding discs }
} Janek Kozicki said: (by the date of Tue, 9 Oct 2007 00:25:50 +0200)
}
} > Richard Scobie said: (by the date of Tue, 09 Oct 2007 08:26:35
} +1300)
} >
} > > No, but you can make a degraded 3 drive array, containing 2 drives and } > > then add the next drive to complete it.
} > >
} > > The array can then be grown (man mdadm, GROW section), to add the } fourth.
} >
} > Oh, good. Thanks, I must've been blind that I missed this.
} > This completely solves my problem.
}
} Uh, actually not :)
}
} My 1st 500 GB drive is full now. When I buy a 2nd one I want to } create a 3-disc degraded array using just 2 discs, one of which } contains unbackupable data.
}
} steps:
} 1. create degraded two-disc RAID5 on 1 new disc
} 2. copy data from old disc to new one
} 3. rebuild the array with old and new discs (now I have 500 GB on 2 discs)
3. Add old disk to new array. Once done RAID5 is redundant.

} 4. GROW this array to a degraded 3 discs RAID5 (so I have 1000 GB on 2 } discs)
4. Buy 3rd disk.
5. Add new 3rd disk to array and grow to 3 disk RAID5 array. Once done, array is redundant.

Repeat 4 and 5 each time you buy a new disk.

I don't think you can grow to a degraded array. I think you must add a new disk first. But I am not sure.

} ...
} 5. when I buy 3rd drive I either grow the array, or just rebuild and } wait with growing until I buy a 4th drive.
}
} Problems at step 4.: 'man mdadm' doesn't tell if it's possible to } grow an array to a degraded array (non existant disc). Is it possible? }
}
} PS: the fact, that degraded array will be unsafe for the data is an } intented motivating factor for buying next drive ;)
}
} --
} Janek Kozicki |
} -
} To unsubscribe from this list: send the line "unsubscribe linux-raid" in } the body of a message to [email protected]
} More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw
In reply to Michael Tokarev
Quoted 6 lines · On Tuesday October 9, [email protected] wrote:

o degraded raid5 isn't really Raid - i.e, it's not any better than a raid0 array, that is, any disk fails => the whole array fails. So instead of creating a degraded raid5 array initially, create smaller one instead, but not degraded, and reshape it when necessary.

Fully agree.

Quoted 10 lines

o reshaping takes time, and for this volume, reshape will take many hours, maybe days, to complete.

o During this reshape time, errors may be fatal to the whole array - while mdadm do have a sense of "critical section", but the whole procedure isn't as much tested as the rest of raid code, I for one will not rely on it, at least for now. For example, a power failure at an "unexpected" moment, or some plain-stupid error in reshape code so that the whole array goes "boom" etc...

While it is true that the resize code is less tested than other code, it is designed to handle a single failure at any time (so a power failure is OK as long as the array is not running degraded), and I have said that if anyone does suffer problems while performing a reshape, I will do my absolute best to get the array functioning and the data safe again.

Quoted 9 lines

o A filesystem on the array has to be resized separately after re{siz,shap}ing the array. And filesystems are different at this point, too - there are various limitations. For example, it's problematic to grow ext[23]fs by large amounts, because when it gets initially created, mke2fs calculates sizes of certain internal data structures based on the device size, and those structures can't be grown significantly, only recreating the filesystem will do the trick.

This isn't entirely true.
For online resizing (while the filesystem is mounted) there are some limitations as you suggest. For offline resizing (while filesystem is not mounted) there are no such limitations.

NeilBrown

#7 lore raw
In reply to Janek Kozicki
Quoted 3 lines · On Tuesday October 9, [email protected] wrote:

Problems at step 4.: 'man mdadm' doesn't tell if it's possible to grow an array to a degraded array (non existant disc). Is it possible?

Why not experiment with loop devices on files and find out?

But yes: you can grow to a degraded array providing you specify a --backup-file.

However I don't recommend it. I would never recommend having a degraded array by design. It should only ever happen due to a failure, and should last only until you can get a replacement rebuilt.

Remember that a degraded raid5 has a greater risk of data loss than a single drive.

Quoted 4 lines

PS: the fact, that degraded array will be unsafe for the data is an intented motivating factor for buying next drive ;)

:-)

NeilBrown

#13 lore raw
In reply to Neil Brown
Quoted 1 line · Neil Brown

On Tuesday October 9, [email protected] wrote:

[]

Quoted 7 lines
Quoted 6 lines

o During this reshape time, errors may be fatal to the whole array - while mdadm do have a sense of "critical section", but the whole procedure isn't as much tested as the rest of raid code, I for one will not rely on it, at least for now. For example, a power failure at an "unexpected" moment, or some plain-stupid error in reshape code so that the whole array goes "boom" etc...

Quoted 6 lines

While it is true that the resize code is less tested than other code, it is designed to handle a single failure at any time (so a power failure is OK as long as the array is not running degraded), and I have said that if anyone does suffer problems while performing a reshape, I will do my absolute best to get the array functioning and the data safe again.

Well... Neil, it's your code, so you trust it - that's ok, I also (tries to) trust my code until someone finds a bug in it.. ;) And I'm a sysadmin (among other things), who's professional property must be a bit of paranoia.. You got the idea ;)

Quoted 9 lines
Quoted 8 lines

o A filesystem on the array has to be resized separately after re{siz,shap}ing the array. And filesystems are different at this point, too - there are various limitations. For example, it's problematic to grow ext[23]fs by large amounts, because when it gets initially created, mke2fs calculates sizes of certain internal data structures based on the device size, and those structures can't be grown significantly, only recreating the filesystem will do the trick.

Quoted 4 lines

This isn't entirely true.
For online resizing (while the filesystem is mounted) there are some limitations as you suggest. For offline resizing (while filesystem is not mounted) there are no such limitations.

There still is - at least for ext[23]. Even offline resizers can't do resizes from any to any size, extfs developers recommend to recreate filesystem anyway if size changes significantly. I'm too lazy to find a reference now, it has been mentioned here on linux-raid at least this year. It's sorta like fat (yea, that ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb, everything is ok, but if you want to go from 501Mb to 1Gb+1, you have to recreate almost all data structures because sizes of all internal fields changes - and here it's much safer to just re-create it from scratch than trying to modify it in place. Sure it's much better for extfs, but the point is still the same.

/mjt

#16 lore raw
In reply to Michael Tokarev

Michael Tokarev said: (by the date of Tue, 09 Oct 2007 02:52:06 +0400)

Quoted 9 lines
Quoted 7 lines · Janek Kozicki

Hello,

Recently I started to use mdadm and I'm very impressed by its capabilities.

I have raid0 (250+250 GB) on my workstation. And I want to have raid5 (4*500 = 1500 GB) on my backup machine.

Quoted 7 lines

Hmm. Are you sure you need that much space on the backup, to start with? Maybe better backup strategy will help to avoid hardware costs? Such as using rsync for backups as discussed on this mailinglist about a month back (rsync is able to keep many ready to use copies of your filesystems but only store files that actually changed since the last backup, thus requiring much less space than many full backups).

yes, exactly. I am using rsnapshot, which is based on rsync and hardlinks. It works exceptionally well - to my knowledge it's the best backup solution I have ever seen. With plugin scripts I am even mounting an lvm-snapshot of the drive being backupped.

from command 'rsnapshot du' I can see how many space is used (but each directory tree is a full backup (made with hardlinks)):

278G    /backup/.sync
454M    /backup/hourly.0/
515M    /backup/hourly.1/
527M    /backup/daily.0/
30G     /backup/daily.1/
21G     /backup/daily.2/
561M    /backup/daily.3/
1.6G    /backup/daily.4/
3.0G    /backup/daily.5/
594M    /backup/daily.6/
1.4G    /backup/weekly.0/
11G     /backup/weekly.1/
9.3G    /backup/weekly.2/
23G     /backup/weekly.3/
33G     /backup/monthly.0/
3.7G    /backup/monthly.1/
415G    total
Quoted 4 lines

It's definitely not possible with raid5. Only option is to create a raid5 array consisting of less drives than it should contain at the end, and reshape it when you get more drives, as others noted in this thread. But do note the following points:

<..snip..>

yes, I am aware of all those problems you listed. The data I'm talking about is already a backup. While the real data is on my workstation (a different linux box - albeit only the newest version of my data). Only losing both of them simultaneoulsy will be catastrophic for me.

So I am inclined to do some experiments with the backup drives configuration, while still doing my best at not losing it. An exercise, you know :)

Quoted 3 lines
Quoted 1 line

is it just a pipe dream?

I'd say it is... ;)

oh well. But I learnt a lot from your answers, thanks a lot!

PS: I'm receiving some mailing list posts twice, anybody knows why? I'm used to mailman but looks like majordomo is being configured in a different way - I cannot find a configure page. (I just subscribed).

Signature · 1 line
Janek Kozicki                                                         |
#8 lore raw
In reply to Neil Brown

Neil Brown said: (by the date of Tue, 9 Oct 2007 13:32:09 +1000)

Quoted 9 lines · On Tuesday October 9, [email protected] wrote:
Quoted 3 lines

Problems at step 4.: 'man mdadm' doesn't tell if it's possible to grow an array to a degraded array (non existant disc). Is it possible?

Why not experiment with loop devices on files and find out?

But yes: you can grow to a degraded array providing you specify a --backup-file.

Thanks! I'll test this on loopback devices :)

Signature · 1 line
Janek Kozicki                                                         |
#9 lore raw
In reply to Neil Brown

Hello Neil ,

Quoted 9 lines · Neil Brown
Quoted 3 lines · On Tuesday October 9, [email protected] wrote:

Problems at step 4.: 'man mdadm' doesn't tell if it's possible to grow an array to a degraded array (non existant disc). Is it possible?

Why not experiment with loop devices on files and find out?

But yes: you can grow to a degraded array providing you specify a --backup-file.

Is there an estimate of how large this file can get ? It's probably a calculation based on disk & array parameters . But I was unable to find a reference on it from the manpage .

Tia , JimL

Quoted 16 lines

However I don't recommend it. I would never recommend having a degraded array by design. It should only ever happen due to a failure, and should last only until you can get a replacement rebuilt.

Remember that a degraded raid5 has a greater risk of data loss than a single drive.

Quoted 4 lines

PS: the fact, that degraded array will be unsafe for the data is an intented motivating factor for buying next drive ;)

:-)

NeilBrown

Signature · 5 lines
+-----------------------------------------------------------------+
| James   W.   Laferriere | System   Techniques | Give me VMS     |
| Network        Engineer | 663  Beaumont  Blvd |  Give me Linux  |
| [email protected] | Pacifica, CA. 94044 |   only  on  AXP |
+-----------------------------------------------------------------+
#10 lore raw
Quoted 12 lines · On Tuesday October 9, [email protected] wrote:

Hello Neil ,

Quoted 9 lines · Neil Brown
Quoted 3 lines · On Tuesday October 9, [email protected] wrote:

Problems at step 4.: 'man mdadm' doesn't tell if it's possible to grow an array to a degraded array (non existant disc). Is it possible?

Why not experiment with loop devices on files and find out?

But yes: you can grow to a degraded array providing you specify a --backup-file.

Review

Quoted 5 lines
(quoted diff) +0 -0
 	Is there an estimate of how large this file can get ?
 	It's probably a calculation based on disk & array parameters .
 	But I was unable to find a reference on it from the manpage .
 
 		Tia ,  JimL

I think:

chunk-size * old-ndisks * new-ndisks * 2 + 1K

It might be smaller than that, but it shouldn't be larger.

NeilBrown

#14 lore raw
In reply to Michael Tokarev
Quoted 12 lines · Michael Tokarev

There still is - at least for ext[23]. Even offline resizers can't do resizes from any to any size, extfs developers recommend to recreate filesystem anyway if size changes significantly. I'm too lazy to find a reference now, it has been mentioned here on linux-raid at least this year. It's sorta like fat (yea, that ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb, everything is ok, but if you want to go from 501Mb to 1Gb+1, you have to recreate almost all data structures because sizes of all internal fields changes - and here it's much safer to just re-create it from scratch than trying to modify it in place. Sure it's much better for extfs, but the point is still the same.

I'll just mention that I once resized a multi-Tera ext3 filesystem and it took 8hours +, a comparable XFS online resize lasted all of 10 seconds!

#15 lore raw
Quoted 15 lines · Louis-David Mitterrand
Quoted 12 lines · Michael Tokarev

There still is - at least for ext[23]. Even offline resizers can't do resizes from any to any size, extfs developers recommend to recreate filesystem anyway if size changes significantly. I'm too lazy to find a reference now, it has been mentioned here on linux-raid at least this year. It's sorta like fat (yea, that ms-dog filesystem) - when you resize it from, say, 501Mb to 999Mb, everything is ok, but if you want to go from 501Mb to 1Gb+1, you have to recreate almost all data structures because sizes of all internal fields changes - and here it's much safer to just re-create it from scratch than trying to modify it in place. Sure it's much better for extfs, but the point is still the same.

Quoted 1 line

I'll just mention that I once resized a multi-Tera ext3 filesystem and it took 8hours +, a comparable XFS online resize lasted all of 10 seconds!

Because of the different way these file systems do things, there is no comparable resize, at least in terms of work to be done. For many systems R/W operations are more common than resize, so the F/S type is selected to optimize that. ;-)

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979