Skip to contentSkip to conversation

Subject: [001/002 ] raid0 reshape

PATCH raz ben yehuda 8 participants 17 messages May 2, 2009 – May 28, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Neil Hello
The bellow is the raid0 grow code.I have decided to fix raid0 and not perform the transformation raid0-raid4-raid0 due to two reasons:
1. raid0 zones. this patch support any zone transformations.
2. Undesired dependency of raid0 over raid4 re-striping code.

The following tests were conducted:
1. various chunk sizes, 4K to 512K. ( mainly in 2.6.27 and 2.6.18 )
2. regrow ( tested on 2.6.27 and 2.6.18 )
3. various super blocks. 0.9 , 1, 1.1 and 1.2 ( mainly in 2.6.27 and 2.6.18 ).
4. support assembling and mounting older raid version ( older kernels and code before patch) after it was grown.

patch passed checkpatch.pl . other than reshaping code i beautified the code. Currently i about to pass this code to our testing team for further tests. Other things to do:
1. Speedup the reshape process.It is too slow.
2. Support for non power 2^n ( page size) chunks.

I will be thankful for your criticism.

Raz

 drivers/md/Kconfig |   13 
 drivers/md/md.c    |    6 
 drivers/md/raid0.c |  711 ++++++++++++++++++++++++++++++++++---------
 drivers/md/raid0.h |    5 
 4 files changed, 590 insertions(+), 145 deletions(-)

---

drivers/md/Kconfig +13 -0
index 36e0675..a9f0ff6 100644
@@ -77,6 +77,19 @@ config MD_RAID0
 
 	  If unsure, say Y.
 
+config MD_RAID0_RESHAPE
+	bool "Support adding drives to a raid-0 array.(EXPERIMENTAL)"
+	depends on MD_RAID0 && EXPERIMENTAL
+	default n
+	---help---
+	  A RAID-0 set can be expanded by adding extra drives. This
+	  requires "restriping" .
+	  You will need mdadm version 2.4.x or later to use this.
+	  The mdadm usage is e.g.
+	       mdadm --grow /dev/md0 --raid-disks=6
+	  Note: The array can only be expanded.
+	  If unsure, say N.
+
 config MD_RAID1
 	tristate "RAID-1 (mirroring) mode"
 	depends on BLK_DEV_MD
drivers/md/md.c +4 -2
index ed5727c..82f57ea 100644
@@ -5707,6 +5707,8 @@ static void status_resync(struct seq_file *seq, mddev_t * mddev)
 		max_blocks = mddev->resync_max_sectors >> 1;
 	else
 		max_blocks = mddev->dev_sectors / 2;
+	if (mddev->level == 0)
+		max_blocks = mddev->array_sectors>>1;
 
 	/*
 	 * Should not happen.
@@ -5915,7 +5917,7 @@ static int md_seq_show(struct seq_file *seq, void *v)
 		if (mddev->pers) {
 			mddev->pers->status(seq, mddev);
 	 		seq_printf(seq, "\n      ");
-			if (mddev->pers->sync_request) {
+			if (mddev->pers->sync_request || !mddev->level) {
 				if (mddev->curr_resync > 2) {
 					status_resync(seq, mddev);
 					seq_printf(seq, "\n      ");
@@ -6146,7 +6148,7 @@ int md_allow_write(mddev_t *mddev)
 		return 0;
 	if (mddev->ro)
 		return 0;
-	if (!mddev->pers->sync_request)
+	if (!mddev->pers->sync_request && mddev->level != 0)
 		return 0;
 
 	spin_lock_irq(&mddev->write_lock);
drivers/md/raid0.c +570 -141

856 lines not on this page · raw message

drivers/md/raid0.h +3 -2
index 824b12e..ff2dca9 100644
@@ -14,9 +14,10 @@ struct raid0_private_data
 {
 	struct strip_zone **hash_table; /* Table of indexes into strip_zone */
 	struct strip_zone *strip_zone;
-	mdk_rdev_t **devlist; /* lists of rdevs, pointed to by strip_zone->dev */
+	mdk_rdev_t **devlist;/* lists of rdevs, pointed to by strip_zone->dev */
 	int nr_strip_zones;
-
+	struct bio *reshape_bi;
+	struct completion wait_reshape;
 	sector_t spacing;
 	int sector_shift; /* shift this before divide by spacing */
 };
#1 lore raw
Quoted 12 lines · On Sunday May 3, [email protected] wrote:

Neil Hello
The bellow is the raid0 grow code.I have decided to fix raid0 and not perform the transformation raid0-raid4-raid0 due to two reasons:
1. raid0 zones. this patch support any zone transformations.
2. Undesired dependency of raid0 over raid4 re-striping code.

The following tests were conducted:
1. various chunk sizes, 4K to 512K. ( mainly in 2.6.27 and 2.6.18 )
2. regrow ( tested on 2.6.27 and 2.6.18 )
3. various super blocks. 0.9 , 1, 1.1 and 1.2 ( mainly in 2.6.27 and 2.6.18 ).
4. support assembling and mounting older raid version ( older kernels and code before patch) after it was grown.

Quoted 7 lines

patch passed checkpatch.pl . other than reshaping code i beautified the code. Currently i about to pass this code to our testing team for further tests. Other things to do:
1. Speedup the reshape process.It is too slow.
2. Support for non power 2^n ( page size) chunks.

I will be thankful for your criticism.

Probably my main criticism at this point is that there is no commentary explaining how it works.
You appear to have chosen to run your own reshape thread rather than providing a "resync" method and make use of the md_do_sync thread which provides speed limiting etc.
Maybe that it a good decision, but as you haven't explained it, it is hard to be sure.

Also, it seems to if an IO request arrives while the reshape is happening, then it fails with -EBUSY.
I don't think that is a good thing.
1/ no filesystem is going to be expecting EBUSY so it could cause
serious problems
2/ if you aren't going to support online reshape so that a device can
be reshaped while it is in use, then there seems to be little point in putting this code in the kernel. Just write a program that runs in userspace which reshapes the array while it is not assembled.

Also, I cannot see any evidence that you checkpoint the reshape at all. So if your machine crashes during the reshape, everything is lost. I do not find this acceptable.

NeilBrown

#2 lore raw
Quoted 21 lines · Neil Brown
Quoted 12 lines · On Sunday May 3, [email protected] wrote:

Neil Hello
The bellow is the raid0 grow code.I have decided to fix raid0 and not perform the transformation raid0-raid4-raid0 due to two reasons:
1. raid0 zones. this patch support any zone transformations.
2. Undesired dependency of raid0 over raid4 re-striping code.

The following tests were conducted:
1. various chunk sizes, 4K to 512K. ( mainly in 2.6.27 and 2.6.18 )
2. regrow ( tested on 2.6.27 and 2.6.18 )
3. various super blocks. 0.9 , 1, 1.1 and 1.2 ( mainly in 2.6.27 and 2.6.18 ).
4. support assembling and mounting older raid version ( older kernels and code before patch) after it was grown.

Quoted 7 lines

patch passed checkpatch.pl . other than reshaping code i beautified the code. Currently i about to pass this code to our testing team for further tests. Other things to do:
1. Speedup the reshape process.It is too slow.
2. Support for non power 2^n ( page size) chunks.

I will be thankful for your criticism.

Quoted 7 lines

Probably my main criticism at this point is that there is no commentary explaining how it works.
You appear to have chosen to run your own reshape thread rather than providing a "resync" method and make use of the md_do_sync thread which provides speed limiting etc.
Maybe that it a good decision, but as you haven't explained it, it is hard to be sure.

You are correct , I will move to md resync; online reshape is essential. I just need to understand the entire md resync processes for that. I am going to
base raid0 reshape code like raid1's, meaning, have two threads, raid0_resync and
raid0d for the writes.
Thank you Neil

Quoted 14 lines

Also, it seems to if an IO request arrives while the reshape is happening, then it fails with -EBUSY.
I don't think that is a good thing.
1/ no filesystem is going to be expecting EBUSY so it could cause serious problems
2/ if you aren't going to support online reshape so that a device can be reshaped while it is in use, then there seems to be little point in putting this code in the kernel.  Just write a program that runs in userspace which reshapes the array while it is not assembled.

Also, I cannot see any evidence that you checkpoint the reshape at all.  So if your machine crashes during the reshape, everything is lost.  I do not find this acceptable.

Quoted 6 lines

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
In reply to raz ben yehuda
Quoted 5 lines · raz ben yehuda

Neil Hello
The bellow is the raid0 grow code.I have decided to fix raid0 and not perform the transformation raid0-raid4-raid0 due to two reasons:
1. raid0 zones. this patch support any zone transformations.
2. Undesired dependency of raid0 over raid4 re-striping code.

Hi Raz,

Can you explain a bit more about why the raid4 approach is undesirable? I think making reshape only available to raid0 arrays where all the members are the same size is a reasonable constraint. We then get the nice benefit of reusing the raid5 reshape infrastructure. In other words I am not convinced that the benefits of reimplementing reshape in raid0 outweigh the costs.

Thanks,
Dan

#4 lore raw
Quoted 7 lines · Dan Williams
Quoted 5 lines · raz ben yehuda

Neil Hello
The bellow is the raid0 grow code.I have decided to fix raid0 and not perform the transformation raid0-raid4-raid0 due to two reasons:
1. raid0 zones. this patch support any zone transformations.
2. Undesired dependency of raid0 over raid4 re-striping code.

Quoted 5 lines

Hi Raz,

Can you explain a bit more about why the raid4 approach is undesirable?  I think making reshape only available to raid0 arrays where all the members are the same size is a reasonable constraint.

1. Because this is the general case. what will you tell a user with two zones ?
2. In my next generation systems, raid0 is on top of 12 raid5s.
A user may choose to expand one of these raid5s separately.
3. The $/Giga ratio will not decline while it should.
Storage lasts years, disks change sizes and formation. Segate
just announced a 2TB disk.
If you start with 4 1TB disks, and in for 4 years you expand this
array 4 times, each time
with a 2TB disk, you end up with 8 TB out of 12 TB. You lose
30% storage. Tell that to your
marketing and tell me when you do that, I want to see :)
4. Number of raid components .
To reach Size X , you need more components with equal size.Having
multiple zones lets you increase
the raid size without being subordinate to an ever increasing
number of disks. you just mirror one of the disks to a bigger disk,
put the new disk back in the array, and reshape with the extra space of the new disk.

Note !
I do not know what is your code base, but if you still have the hash spacing, you might
have two zones even if the underlying components are the same in size. So, in this case your decision if to support or not support the raid0 reshape is not just nb_zone > 1.

Other than that, if you do this work, it be will nice to have my back covered sooner than expected.(as long as you will NOT reach for your revolver.... :) )

Quoted 10 lines

We then get the nice benefit of reusing the raid5 reshape infrastructure.  In other words I am not convinced that the benefits of reimplementing reshape in raid0 outweigh the costs. Thanks,
Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
In reply to Dan Williams
Quoted 7 lines · On Tuesday May 19, [email protected] wrote:
Quoted 5 lines · raz ben yehuda

Neil Hello
The bellow is the raid0 grow code.I have decided to fix raid0 and not perform the transformation raid0-raid4-raid0 due to two reasons:
1. raid0 zones. this patch support any zone transformations.
2. Undesired dependency of raid0 over raid4 re-striping code.

Quoted 8 lines

Hi Raz,

Can you explain a bit more about why the raid4 approach is undesirable? I think making reshape only available to raid0 arrays where all the members are the same size is a reasonable constraint. We then get the nice benefit of reusing the raid5 reshape infrastructure. In other words I am not convinced that the benefits of reimplementing reshape in raid0 outweigh the costs.

I've been thinking about this too... Is it something we really want to do?

My thoughts include:

- I don't like special cases - it would be nice to support reshape on all arrays, even RAID0 with different sizes devices.
- Anyone who does this with a raid0 made of simple drives is asking for trouble. But a RAID0 over a bunch of RAID5 or RAID1 might make sense.
- Maybe we should support different sized drives in RAID4. As long as the parity drive is as big as the largest data drive it could be made to work. Similarly hot spares would need to be big, but you could have 2 hot spares and take the smallest one that is big enough.
If a drive in the RAID4+ (or is it is the thing called NORAID?) failed and was replaced with a bigger drive, it would be cool to be able to incorporate that extra space into the array.

If we did all that, then the 0->4->0 conversion could make use of the same code.
- Surely RAID0 is (like LVM) just a legacy idea until we get sensible file systems that actually understand multiple devices and do all this stuff for you are a more sensible level - so why are we busting a gut(*) to make RAID0 work well?? Answer is of course that no-one has made a sensible file system yet. (well... maybe zfs or btrfs, not sure)
- If you read the DDF spec carefully, you find there is a secondary raid level which stripes over heterogeneous arrays a different way. You divide every primary array up into N chunks, so the chunk sizes are different on different arrays. Then you make a secondary array by striping over those chunks.
So e.g. you might have a 4Gig RAID5 and a 1GIG RAID1. So the stripes array on top of these could take 4Meg from the RAID5, then 1Meg from the RAID1, then another 4 from the RAID5 etc.
So we want to support that? And would we want to reshape such a thing??

So: lots of thoughts, some pointing in different directions. But I'm not against reshape code appearing in RAID0 providing it is well designed, maintainable, reliable, and doesn't slow down normal RAID0 processing. I suspect we can get there.

NeilBrown

* is that an Australian term??? not sure. http://www.wordwebonline.com/en/BUSTAGUT

#6 lore raw

OT: busting a gut (was Re: Subject: [001/002 ] raid0 reshape)

On 21/05/2009 12:48, Neil Brown wrote:
[...]

Quoted 1 line

why are we busting a gut(*) to make RAID0 work well??

[...]

Quoted 1 line

* is that an Australian term??? not sure. http://www.wordwebonline.com/en/BUSTAGUT

No, or at least not exclusively; I'm familiar with it being used in the UK.

Cheers,

John.

#7 lore raw
In reply to Neil Brown
Quoted 55 lines · Neil Brown
Quoted 7 lines · On Tuesday May 19, [email protected] wrote:
Quoted 5 lines · raz ben yehuda

Neil Hello
The bellow is the raid0 grow code.I have decided to fix raid0 and not perform the transformation raid0-raid4-raid0 due to two reasons:
1. raid0 zones. this patch support any zone transformations.
2. Undesired dependency of raid0 over raid4 re-striping code.

Quoted 8 lines

Hi Raz,

Can you explain a bit more about why the raid4 approach is undesirable?  I think making reshape only available to raid0 arrays where all the members are the same size is a reasonable constraint. We then get the nice benefit of reusing the raid5 reshape infrastructure.  In other words I am not convinced that the benefits of reimplementing reshape in raid0 outweigh the costs.

I've been thinking about this too... Is it something we really want to do?

My thoughts include:

 - I don't like special cases - it would be nice to support reshape on all arrays, even RAID0 with different sizes devices.
 - Anyone who does this with a raid0 made of simple drives is asking for trouble.  But a RAID0 over a bunch of RAID5 or RAID1 might make sense.
 - Maybe we should support different sized drives in RAID4.  As long as the parity drive is as big as the largest data drive it could be made to work.  Similarly hot spares would need to be big, but you could have 2 hot spares and take the smallest one that is big enough.
  If a drive in the RAID4+ (or is it is the thing called NORAID?) failed and was replaced with a bigger drive, it would be cool to be able to incorporate that extra space into the array.

  If we did all that, then the 0->4->0 conversion could make use of the same code.
 - Surely RAID0 is (like LVM) just a legacy idea until we get sensible file systems that actually understand multiple devices and do all this stuff for you are a more sensible level - so why are we busting a gut(*) to make RAID0 work well??  Answer is of course that no-one has made a sensible file system yet. (well... maybe zfs or btrfs, not sure)
 - If you read the DDF spec carefully, you find there is a secondary raid level which stripes over heterogeneous arrays a different way. You divide every primary array up into N chunks, so the chunk sizes are different on different arrays.  Then you make a secondary array by striping over those chunks.
  So e.g. you might have a 4Gig RAID5 and a 1GIG RAID1.  So the stripes array on top of these could take 4Meg from the RAID5, then 1Meg from the RAID1, then another 4 from the RAID5 etc.
  So we want to support that?  And would we want to reshape such a thing??

Quoted 6 lines

So: lots of thoughts, some pointing in different directions. But I'm not against reshape code appearing in RAID0 providing it is well designed, maintainable, reliable, and doesn't slow down normal RAID0 processing.  I suspect we can get there.

NeilBrown

This may be in a FAQ / wiki somewhere, but is the long range plan of mdraid to stick to the formal "raid levels" or to implement "raid equivalent levels".

I am a big fan of the raid equivalent concept and would love to see mdraid moving in that direction.

==> What I mean by raid equivalent levels

More and more arrays allow the user to simply say "give me a 100 GB logical volume with Raid 5 equivalent protection. The array then looks at the drives it has available and puts together the necessary pieces. As drives are added, removed it moves the data around under its own control, but maintains the raid equivalent protection.

Especially when working with dozens of drives and lots of logical volumes it makes life much easier. Admittedly it may come at a cost of not being able to specify raid levels with the specificity that mdraid currently allows.

==>

The reason I ask if this is the goal is that doing so may factor into decisions about how reshaping is implemented.

Greg

Signature · 14 lines
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#8 lore raw
Quoted 13 lines · Greg Freemyer

==> What I mean by raid equivalent levels

More and more arrays allow the user to simply say "give me a 100 GB logical volume with Raid 5 equivalent protection. The array then looks at the drives it has available and puts together the necessary pieces. As drives are added, removed it moves the data around under its own control, but maintains the raid equivalent protection.

Especially when working with dozens of drives and lots of logical volumes it makes life much easier. Admittedly it may come at a cost of not being able to specify raid levels with the specificity that mdraid currently allows.

Quoted 6 lines

==>

The reason I ask if this is the goal is that doing so may factor into decisions about how reshaping is implemented.

Greg

That really seems to scream for LVM to support more raid levels. It already has linear, raid0 and raid1 support (although I have no idea how device mapper raid1 compares to md raid1).

Those should be fleshed out more and also support raid 4/5/6 for what you ask.

MfG
Goswin

#9 lore raw

It is not clear to me why Linux has both LVM and md,waste of development effort to my opinion. Adding to that brtfs/zfs reaching mainline, Linux will have 3-4 volume managers to maintain. why not join hands and come up with a single unify system?

Quoted 29 lines · Goswin von Brederlow
Quoted 13 lines · Greg Freemyer

==> What I mean by raid equivalent levels

More and more arrays allow the user to simply say "give me a 100 GB logical volume with Raid 5 equivalent protection.  The array then looks at the drives it has available and puts together the necessary pieces.  As drives are added, removed it moves the data around under its own control, but maintains the raid equivalent protection.

Especially when working with dozens of drives and lots of logical volumes it makes life much easier.  Admittedly it may come at a cost of not being able to specify raid levels with the specificity that mdraid currently allows.

Quoted 6 lines

==>

The reason I ask if this is the goal is that doing so may factor into decisions about how reshaping is implemented.

Greg

That really seems to scream for LVM to support more raid levels. It already has linear, raid0 and raid1 support (although I have no idea how device mapper raid1 compares to md raid1).

Those should be fleshed out more and also support raid 4/5/6 for what you ask.

Quoted 7 lines

MfG
       Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw
Quoted 5 lines · Raz

It is not clear to me why Linux has both LVM and md,waste of development effort to my opinion. Adding to that brtfs/zfs reaching mainline, Linux will have 3-4 volume managers to maintain. why not join hands and come up with a single unify system?

Linux is about choice? One size doesn't fit all?

I see no reason for raid anything in LVM, it's a duplication of effort. By the same token, I think building everything into the file system, while it seems nice, means that you lose the flexibility of being able to control the devices, the raid behavior, and the allocation, each independently. I'll be the first to admit that I occasionally abuse that flexibility (story coming one of these days), but it's there.

As long as there's a tool to help the novice put the pieces together without in-depth technical expertise, I don't think limiting options to just one is desirable at all.

Quoted 32 lines · Goswin von Brederlow
Quoted 31 lines
Quoted 13 lines · Greg Freemyer

==> What I mean by raid equivalent levels

More and more arrays allow the user to simply say "give me a 100 GB logical volume with Raid 5 equivalent protection. The array then looks at the drives it has available and puts together the necessary pieces. As drives are added, removed it moves the data around under its own control, but maintains the raid equivalent protection.

Especially when working with dozens of drives and lots of logical volumes it makes life much easier. Admittedly it may come at a cost of not being able to specify raid levels with the specificity that mdraid currently allows.

Quoted 7 lines

==>

The reason I ask if this is the goal is that doing so may factor into decisions about how reshaping is implemented.

Greg

That really seems to scream for LVM to support more raid levels. It already has linear, raid0 and raid1 support (although I have no idea how device mapper raid1 compares to md raid1).

Those should be fleshed out more and also support raid 4/5/6 for what you ask.

Signature · 6 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc

"You are disgraced professional losers. And by the way, give us our money back."
   - Representative Earl Pomeroy,  Democrat of North Dakota
on the A.I.G. executives who were paid bonuses  after a federal bailout.
#11 lore raw
In reply to Goswin von Brederlow
Quoted 17 lines · On Monday May 25, [email protected] wrote:
Quoted 15 lines · Greg Freemyer

==> What I mean by raid equivalent levels

More and more arrays allow the user to simply say "give me a 100 GB logical volume with Raid 5 equivalent protection. The array then looks at the drives it has available and puts together the necessary pieces. As drives are added, removed it moves the data around under its own control, but maintains the raid equivalent protection.

Especially when working with dozens of drives and lots of logical volumes it makes life much easier. Admittedly it may come at a cost of not being able to specify raid levels with the specificity that mdraid currently allows.

==>

Quoted 8 lines
Quoted 4 lines

The reason I ask if this is the goal is that doing so may factor into decisions about how reshaping is implemented.

Greg

That really seems to scream for LVM to support more raid levels. It already has linear, raid0 and raid1 support (although I have no idea how device mapper raid1 compares to md raid1).

Note that LVM (a suite of user-space tools) could conceivably use md/raid1, md/raid5 etc. The functionality doesn't have to go in dm.

Neil

Quoted 6 lines

Those should be fleshed out more and also support raid 4/5/6 for what you ask.

MfG
Goswin

#12 lore raw
Quoted 9 lines · Neil Brown
Quoted 3 lines · On Monday May 25, [email protected] wrote:

That really seems to scream for LVM to support more raid levels. It already has linear, raid0 and raid1 support (although I have no idea how device mapper raid1 compares to md raid1).

Note that LVM (a suite of user-space tools) could conceivably use md/raid1, md/raid5 etc. The functionality doesn't have to go in dm.

Neil

How would you do this? Worst case you can have a LV made up of totaly non linear PEs, meaning lots of 4MB (default PE size) big chunks in random order on random disks.

Do you create a raid1/5 for each stripe? You surely run out of md devices.

Create dm mappings for all stripe 0s, stripe 1s, stripe 2s, ... and then a raid1/5 over those stripe devices?

What if the LV has segments with different raid configurations (number of disks in a stripe or even different levels)? Create a raid for each segment and then a dm mapping for a linear raid?

You can get a flood of intermediate devices there. A /proc/mdstat with 200 entries would be horrible. iostat output would be totaly useless. ...

MfG
Goswin

#13 lore raw
Quoted 10 lines · On Tuesday May 26, [email protected] wrote:
Quoted 8 lines · Neil Brown
Quoted 3 lines · On Monday May 25, [email protected] wrote:

That really seems to scream for LVM to support more raid levels. It already has linear, raid0 and raid1 support (although I have no idea how device mapper raid1 compares to md raid1).

Note that LVM (a suite of user-space tools) could conceivably use md/raid1, md/raid5 etc. The functionality doesn't have to go in dm.

Quoted 8 lines
Quoted 1 line

Neil

How would you do this? Worst case you can have a LV made up of totaly non linear PEs, meaning lots of 4MB (default PE size) big chunks in random order on random disks.

Do you create a raid1/5 for each stripe? You surely run out of md devices.

We have 2^21 md devices easily (I think that is the number) and it wouldn't be hard to have more if that were an issue.

Quoted 3 lines

Create dm mappings for all stripe 0s, stripe 1s, stripe 2s, ... and then a raid1/5 over those stripe devices?

That might be an option.

Quoted 5 lines

What if the LV has segments with different raid configurations (number of disks in a stripe or even different levels)? Create a raid for each segment and then a dm mapping for a linear raid?

Yes.

Quoted 5 lines

You can get a flood of intermediate devices there. A /proc/mdstat with 200 entries would be horrible. iostat output would be totaly useless. ...

Yep, these would be interesting problems to solve. /proc/mdstat is a bit of a wart on the design - getting the entry in /proc/mdstat optional might be a good idea.

As for iostat - where does it get info from ? /proc/partitions? /proc/diskinfo? Maybe /sys/block?
Either way, we could probably find a way to say "this block device is 'hidden'" .

If you want to be able to slice and dice lot of mini-raid arrays into an LVM system, then whatever way you implement it you will need to be keeping track of all those bits. I think it makes most sense to use the "block device" as the common abstraction, then if we start finding
issues: solve them. That way the solutions become available for others to use in ways we hadn't expected.

Quoted 2 lines

MfG
Goswin

NeilBrown

#14 lore raw
Quoted 38 lines · Neil Brown
Quoted 10 lines · On Tuesday May 26, [email protected] wrote:
Quoted 8 lines · Neil Brown
Quoted 3 lines · On Monday May 25, [email protected] wrote:

That really seems to scream for LVM to support more raid levels. It already has linear, raid0 and raid1 support (although I have no idea how device mapper raid1 compares to md raid1).

Note that LVM (a suite of user-space tools) could conceivably use md/raid1, md/raid5 etc. The functionality doesn't have to go in dm.

Quoted 8 lines
Quoted 1 line

Neil

How would you do this? Worst case you can have a LV made up of totaly non linear PEs, meaning lots of 4MB (default PE size) big chunks in random order on random disks.

Do you create a raid1/5 for each stripe? You surely run out of md devices.

We have 2^21 md devices easily (I think that is the number) and it wouldn't be hard to have more if that were an issue.

Quoted 3 lines

Create dm mappings for all stripe 0s, stripe 1s, stripe 2s, ... and then a raid1/5 over those stripe devices?

That might be an option.

Quoted 5 lines

What if the LV has segments with different raid configurations (number of disks in a stripe or even different levels)? Create a raid for each segment and then a dm mapping for a linear raid?

Yes.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

You can get a flood of intermediate devices there. A /proc/mdstat with 200 entries would be horrible. iostat output would be totaly useless. ...

Yep, these would be interesting problems to solve. /proc/mdstat is a bit of a wart on the design - getting the entry in /proc/mdstat optional might be a good idea.

Resyncing in a way that uses parallelism without using a physical devices twice would also be difficult without merging all those layers into one or peaking through them. The raid could doesn't see what physical devices are in a device-mapper device and so on.

Plus I do want ONE entry in /proc/mdstat (or equivalent) to see how a resync is going. Just not 200. So it is not just about hiding but also about showing something sensible.

Quoted 4 lines

As for iostat - where does it get info from ? /proc/partitions? /proc/diskinfo? Maybe /sys/block?
Either way, we could probably find a way to say "this block device is 'hidden'" .

One of those places.

Quoted 6 lines

If you want to be able to slice and dice lot of mini-raid arrays into an LVM system, then whatever way you implement it you will need to be keeping track of all those bits. I think it makes most sense to use the "block device" as the common abstraction, then if we start finding
issues: solve them. That way the solutions become available for others to use in ways we hadn't expected.

I think the device mapper tables should suffice. They are perfect for slice and dice operations. This should realy sidestep the block device overhead (alloc major/minor, send event, not runtime overhead) and combine status of many slices into a combined status.

I see one problem though for converting md code to dm code: The metadata. In LVM every PE is basically independent and can be moved around at will. So the raid code must be able to split and merge raid devices on a PE granularity at least. Specifically the dirty/clean informations and serial counts are tricky.

There could be 2 options:

1) Put a little bit of metadata at the start of every PE. The first block of each PE could also hold an internal bitmap for that PE and not just a few meta infos and the clean/dirty byte. For internal bitmaps this might be optimal as it would garanty short seeks to reach the bits.

2) Have detached metadata. Md already has detached bitmaps. Think of it as a raid without metadata but with external bitmap.

Quoted 2 lines
Quoted 2 lines

MfG
Goswin

MfG
Goswin

#15 lore raw
In reply to Neil Brown
Quoted 8 lines · Neil Brown
Quoted 1 line · On Tuesday May 19, [email protected] wrote:

On Sat, May 2, 2009 at 2:46 PM, raz ben yehuda <[email protected]> wrote:

 - Surely RAID0 is (like LVM) just a legacy idea until we get sensible file systems that actually understand multiple devices and do all this stuff for you are a more sensible level - so why are we busting a gut(*) to make RAID0 work well??  Answer is of course that no-one has made a sensible file system yet. (well... maybe zfs or btrfs, not sure)

"Compatibility" is another cause of the abdominal pressure. See the single drive to raid0 migrations supported by the Windows driver:
http://www.intel.com/support/chipsets/imsm/sb/cs-020674.htm

Quoted 4 lines

So: lots of thoughts, some pointing in different directions. But I'm not against reshape code appearing in RAID0 providing it is well designed, maintainable, reliable, and doesn't slow down normal RAID0 processing.  I suspect we can get there.

Nod.

Thanks,
Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#16 lore raw
In reply to Neil Brown
Quoted 36 lines · Neil Brown
Quoted 7 lines · On Tuesday May 19, [email protected] wrote:
Quoted 5 lines · raz ben yehuda

Neil Hello
The bellow is the raid0 grow code.I have decided to fix raid0 and not perform the transformation raid0-raid4-raid0 due to two reasons:
1. raid0 zones. this patch support any zone transformations.
2. Undesired dependency of raid0 over raid4 re-striping code.

Quoted 8 lines

Hi Raz,

Can you explain a bit more about why the raid4 approach is undesirable?  I think making reshape only available to raid0 arrays where all the members are the same size is a reasonable constraint. We then get the nice benefit of reusing the raid5 reshape infrastructure.  In other words I am not convinced that the benefits of reimplementing reshape in raid0 outweigh the costs.

I've been thinking about this too... Is it something we really want to do?

My thoughts include:

 - I don't like special cases - it would be nice to support reshape on all arrays, even RAID0 with different sizes devices.
 - Anyone who does this with a raid0 made of simple drives is asking for trouble.  But a RAID0 over a bunch of RAID5 or RAID1 might make sense.
 - Maybe we should support different sized drives in RAID4.  As long as the parity drive is as big as the largest data drive it could be made to work.  Similarly hot spares would need to be big, but you could have 2 hot spares and take the smallest one that is big enough.
  If a drive in the RAID4+ (or is it is the thing called NORAID?) failed and was replaced with a bigger drive, it would be cool to be able to incorporate that extra space into the array.

Quoted 8 lines

  If we did all that, then the 0->4->0 conversion could make use of the same code.
 - Surely RAID0 is (like LVM) just a legacy idea until we get sensible file systems that actually understand multiple devices and do all this stuff for you are a more sensible level - so why are we busting a gut(*) to make RAID0 work well??  Answer is of course that no-one has made a sensible file system yet. (well... maybe zfs or btrfs, not sure)

there is pvfs2 that stripes in a file level. though without redundancy. but i do
consider pvfs2 as a profesional file system.

Quoted 10 lines

 - If you read the DDF spec carefully, you find there is a secondary raid level which stripes over heterogeneous arrays a different way. You divide every primary array up into N chunks, so the chunk sizes are different on different arrays.  Then you make a secondary array by striping over those chunks.
  So e.g. you might have a 4Gig RAID5 and a 1GIG RAID1.  So the stripes array on top of these could take 4Meg from the RAID5, then 1Meg from the RAID1, then another 4 from the RAID5 etc.
  So we want to support that?  And would we want to reshape such a thing??

i want;but what i do wonder is how raid awared file systems are to be tuned. chunk=stripe ?

Quoted 10 lines

So: lots of thoughts, some pointing in different directions. But I'm not against reshape code appearing in RAID0 providing it is well designed, maintainable, reliable, and doesn't slow down normal RAID0 processing.  I suspect we can get there.

NeilBrown

Quoted 6 lines

* is that an Australian term??? not sure.  http://www.wordwebonline.com/en/BUSTAGUT
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html