Skip to contentSkip to conversation

how to synchronize two devices (RAID-1, but not really?)

DISCUSSION Tomasz Chmielewski 7 participants 15 messages May 15, 2007 – May 16, 2007 MD RAID

lore
Share & tools
lore
#0 lore raw

I have a RAID-10 setup of four 400 GB HDDs. As the data grows by several GBs a day, I want to migrate it somehow to RAID-5 on separate disks in a separate machine.

Which would be easy, if I didn't have to do it online, without stopping any services.

M1 - machine 1, RAID-10
M2 - machine 2, RAID-5

My first idea was to copy the data with rsync two or three times (because the files change, I would stop the services for the last run) - which turned out to be totally unrealistic - I started rsync process two days ago, and it still calculates files to be copied (over 100 million files, with hardlinks etc.).

So my second idea is to do some sort of RAID-1-over-iSCSI of these two arrays - I don't care if it's gonna take long, what's important, I don't want to stop the services.

So let's say M2 exports RAID-5 md to M1 over iSCSI, and M2's RAID-5 is available to M1 as /dev/sdr.

Now, is there a way I can synchronize the contents of RAID-10, or /dev/md10, with the contents of RAID-5, or /dev/sdr, when /dev/sdr is bigger then /dev/md10, and /dev/md10 has to be synchronized on /dev/sdr, not the way around (I would expand the filesystem later with ext3online)?

Perhaps "dd" tool would be fine if the data didn't change.

RAID-1 would be great, as it tracks changes, but I'd have to create RAID-1 over: /dev/md10 and /dev/sdr. Wouldn't it destroy the contents of /dev/md10?

Ideas how to synchronize the contents of two devices (device1 -> device2)?

Signature · 1 line
Tomasz Chmielewski
#1 lore raw
Quoted 8 lines · Tomasz Chmielewski

I have a RAID-10 setup of four 400 GB HDDs. As the data grows by several GBs a day, I want to migrate it somehow to RAID-5 on separate disks in a separate machine.

Which would be easy, if I didn't have to do it online, without stopping any services.

Your /dev/md10 - what is directly on top of it? LVM? XFS? EXT3?

#2 lore raw
Quoted 11 lines · Peter Rabbitson
Quoted 8 lines · Tomasz Chmielewski

I have a RAID-10 setup of four 400 GB HDDs. As the data grows by several GBs a day, I want to migrate it somehow to RAID-5 on separate disks in a separate machine.

Which would be easy, if I didn't have to do it online, without stopping any services.

Your /dev/md10 - what is directly on top of it? LVM? XFS? EXT3?

Good point. I don't want to copy the whole RAID-10.
I want to copy only one LVM-2 volume (which is like 90% of that RAID-10, anyway).

So I want to synchronize /dev/LVM2/my-volume (ext3) with /dev/sdr (now empty; bigger than /dev/LVM2/my-volume).

(sda2, sdb2, sdc2, sdd2) -> RAID-10 -> LVM-2 -> my volume -> ext3

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#3 lore raw
Quoted 18 lines · Tomasz Chmielewski
Quoted 11 lines · Peter Rabbitson
Quoted 8 lines · Tomasz Chmielewski

I have a RAID-10 setup of four 400 GB HDDs. As the data grows by several GBs a day, I want to migrate it somehow to RAID-5 on separate disks in a separate machine.

Which would be easy, if I didn't have to do it online, without stopping any services.

Your /dev/md10 - what is directly on top of it? LVM? XFS? EXT3?

Good point. I don't want to copy the whole RAID-10.
I want to copy only one LVM-2 volume (which is like 90% of that RAID-10, anyway).

Quoted 7 lines

So I want to synchronize /dev/LVM2/my-volume (ext3) with /dev/sdr (now empty; bigger than /dev/LVM2/my-volume).

(sda2, sdb2, sdc2, sdd2) -> RAID-10 -> LVM-2 -> my volume -> ext3

I've not used iSCSI but I wonder about using nbd : network block device

Use nbd to export /dev/md5 from machine 2.
Import /dev/nbd0 on machine 1.
Add nbd0 to the VG on machine 1
pvmove the data from /dev/md10 to /dev/nbd0 (ie the md5 on machine2 via nbd) remove /dev/md10 from the VG.
The VG should now exist only on /dev/nbd0 on machine 2
stop the services and lvm on machine 1
start the lvm and services on machine 2.

I'd suggest testing this first <grin>.

David

#4 lore raw

David Greaves schrieb:

(...)

Quoted 9 lines
Quoted 7 lines

So I want to synchronize /dev/LVM2/my-volume (ext3) with /dev/sdr (now empty; bigger than /dev/LVM2/my-volume).

(sda2, sdb2, sdc2, sdd2) -> RAID-10 -> LVM-2 -> my volume -> ext3

Quoted 7 lines

I've not used iSCSI but I wonder about using nbd : network block device

Use nbd to export /dev/md5 from machine 2.
Import /dev/nbd0 on machine 1.
Add nbd0 to the VG on machine 1
pvmove the data from /dev/md10 to /dev/nbd0 (ie the md5 on machine2 via nbd) remove /dev/md10 from the VG.

Hmm, maybe using LVM's mirror would help me here:

CONFIG_DM_MIRROR:

Allow volume managers to mirror logical volumes, also

needed for live data migration tools such as 'pvmove'.

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#5 lore raw
In reply to David Greaves
Quoted 43 lines · David Greaves
Quoted 20 lines · Tomasz Chmielewski
Quoted 6 lines · Peter Rabbitson
Quoted 4 lines · Tomasz Chmielewski

I have a RAID-10 setup of four 400 GB HDDs. As the data grows by several GBs a day, I want to migrate it somehow to RAID-5 on separate disks in a separate machine.

Quoted 7 lines
Quoted 5 lines

Which would be easy, if I didn't have to do it online, without stopping any services.

Your /dev/md10 - what is directly on top of it? LVM? XFS? EXT3?

Good point. I don't want to copy the whole RAID-10.
I want to copy only one LVM-2 volume (which is like 90% of that RAID-10, anyway).

Quoted 8 lines

So I want to synchronize /dev/LVM2/my-volume (ext3) with /dev/sdr (now empty; bigger than /dev/LVM2/my-volume).

(sda2, sdb2, sdc2, sdd2) -> RAID-10 -> LVM-2 -> my volume -> ext3

I've not used iSCSI but I wonder about using nbd : network block device

Use nbd to export /dev/md5 from machine 2.
Import /dev/nbd0 on machine 1.
Add nbd0 to the VG on machine 1
pvmove the data from /dev/md10 to /dev/nbd0 (ie the md5 on machine2 via nbd) remove /dev/md10 from the VG.
The VG should now exist only on /dev/nbd0 on machine 2
stop the services and lvm on machine 1
start the lvm and services on machine 2.

Quoted 2 lines

I'd suggest testing this first <grin>.

Posts to the kernel mailing list indicate that people, including me, have had issues with this since 2.6.10-rc versions. I got no "me too" responses, so I figured something have changed, although yesterday someone reported a hard crash using nbd.

Test it well, but it's the ideal solution, combined with a bitmap, for getting in sync.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
#6 lore raw
In reply to Tomasz Chmielewski
Quoted 11 lines · Tomasz Chmielewski
Quoted 10 lines · Peter Rabbitson
Quoted 8 lines · Tomasz Chmielewski

I have a RAID-10 setup of four 400 GB HDDs. As the data grows by several GBs a day, I want to migrate it somehow to RAID-5 on separate disks in a separate machine.

Which would be easy, if I didn't have to do it online, without stopping any services.

Quoted 8 lines
Quoted 1 line

Your /dev/md10 - what is directly on top of it? LVM? XFS? EXT3?

Good point. I don't want to copy the whole RAID-10.
I want to copy only one LVM-2 volume (which is like 90% of that RAID-10, anyway).

So I want to synchronize /dev/LVM2/my-volume (ext3) with /dev/sdr (now empty; bigger than /dev/LVM2/my-volume).

[...]

This actually makes it quite a bit easier, assuming you have some reasonable amount of unallocated space in your LVM volume group. For the purpose of the commands below, I'm going to call that amount <FREESPACE>. Do the following (as root) on the machine with the LVM2 VG:

lvcreate -L <FREESPACE> -p r -n copy-me -s LVM2/my-volume

This now gives you an unchanging, read-only device /dev/LVM2/copy-me which you can then dd over netcat or the like to an equal-sized volume on your destination RAID. Let that finish (I'm sure it will take some time).

Once you're done with that step, run lvdisplay LVM2/copy-me and take a look at the "Allocated to snapshot" line. That is a percentage of <FREESPACE> that is different between the current live volume and the snapshot. Do the multiplication and figure out how much storage is actually different, then get rid of the snapshot with:

lvremove -f LVM2/copy-me

You should know how long you can reasonably have things down. To get an idea of how long X amount of data will take to rsync, make another lvm snapshot with the lvcreate commandline above and rsync between the two actual devices (i.e. not the filesystems, but /dev/LVM2/copy-me and the destination volume's block device).

You now have to decide if the amount of data difference will take longer than the downtime you can accept. If so, repeat the process of snapshot, rsync, lvdisplay/calculate until it gets to an acceptable level or you reach a point of diminishing returns. From there, you take down your services and such, unmount the live LVM volume, rsync one more time, and bring stuff up on the new RAID.

Depending on how many loops you have to go through this may take some time and effort, but it will minimize your downtime (which, it seems, is your goal). Be sure to lvremove the snapshot every time, since you are reusing the space allocated to it for the next snapshot. Also note that if you are writing to the live partition faster than you can rsync it, I see no way to avoid significant downtime.

Quoted 1 line

Tomasz Chmielewski

--Greg

#7 lore raw
In reply to Tomasz Chmielewski
Quoted 10 lines · Tomasz Chmielewski

I have a RAID-10 setup of four 400 GB HDDs. As the data grows by several GBs a day, I want to migrate it somehow to RAID-5 on separate disks in a separate machine.

Which would be easy, if I didn't have to do it online, without stopping any services.

M1 - machine 1, RAID-10
M2 - machine 2, RAID-5

My first idea was to copy the data with rsync two or three times (because the files change, I would stop the services for the last run) - which turned out to be totally unrealistic - I started rsync process two days ago, and it still calculates files to be copied (over 100 million files, with hardlinks etc.).

I have recent experience of copying up to 1TB parititions to offsite backup servers via rsync (and a 10Mb line), and it can be done.

You do need a recent version of rsync and a lot of memory in both servers. I'm really surprised it takes days to do this on your server - although maybe the source server really is busy? I was doing in over about 2-3 hours on the servers I was working with. (Several "smaller" ones backing up to one 6TB box) The first copy is always the longest, but if after 2 days it's not even started to copy the files, then it's gonig to be slow. Lots of hardlinks will really slow things down though (and cause memory to grow), as it has to find out each filename linked to each file. I've not looked too closely into it, but I suspect it's as efficient as a bubble sort due to what it has to do. Lots of memory will help if it can keep everything cached as well as letting rsync build up it's data without swapping.

Quoted 1 line

Ideas how to synchronize the contents of two devices (device1 -> device2)?

Look into the speed of the processors too - rsync will use ssh by default - which will incur a CPU overheard to do the encryption. If encryption isn't an issue, then look into using rsh rather than ssh. (Although making a modern system let you login as root with rsh without a password is sometimes "challenging" :)

Another solution might be to use tar, but this is good for snapshotting rather than applying incremental updates. something like

tar cf - /bigVolume | rsh remoteServer -c 'cd target ; tar xfB -'

or something. (it's years since I've done it this way, so check!) You might want to set blocking factors which might improve network throughput. (And you never know, a modern tar might be able to do incremental changes)

Gordon

#8 lore raw
Quoted 11 lines · Gordon Henderson
Quoted 9 lines · Tomasz Chmielewski

I have a RAID-10 setup of four 400 GB HDDs. As the data grows by several GBs a day, I want to migrate it somehow to RAID-5 on separate disks in a separate machine.

Which would be easy, if I didn't have to do it online, without stopping any services.

M1 - machine 1, RAID-10
M2 - machine 2, RAID-5

Quoted 7 lines
Quoted 3 lines

My first idea was to copy the data with rsync two or three times (because the files change, I would stop the services for the last run)
- which turned out to be totally unrealistic - I started rsync process
two days ago, and it still calculates files to be copied (over 100 million files, with hardlinks etc.).

I have recent experience of copying up to 1TB parititions to offsite backup servers via rsync (and a 10Mb line), and it can be done.

You do need a recent version of rsync and a lot of memory in both servers. I'm really surprised it takes days to do this on your server - although maybe the source server really is busy?

Yes, the server is really busy and reads/writes a lot - that's one. Two, it's being done over iSCSI in a pretty complicated scenario:

                    100 Mbit
M1 (RAID-10 storage) <--> very busy virtual machine in Xen
                           /
                          / 1 Gbit
                         /
     machine with rsync process <--> M2 (RAID-5 storage)
                               1 Gbit

So there is a slow point of just 100 Mbit to access the storage, which is normally fully filled anyway.

rsync, even 3.0, just doesn't fit here - it has to be done in sort of a "mirror" way.

Signature · 2 lines
Tomasz Chmielewski
htp://wpkg.org
#9 lore raw
In reply to Tomasz Chmielewski
Quoted 5 lines · On Tuesday May 15, [email protected] wrote:

Now, is there a way I can synchronize the contents of RAID-10, or /dev/md10, with the contents of RAID-5, or /dev/sdr, when /dev/sdr is bigger then /dev/md10, and /dev/md10 has to be synchronized on /dev/sdr, not the way around (I would expand the filesystem later with ext3online)?

Yes, you can do this.

You want a raid1 with no metadata, an external bitmap, and a write-mostly, write-behind device. Simple?

No metadata means no superblock is stored on the device. You get to use all of it for data, which is good if it is already all in use for data.
The down-side of not having a superblock is that you have to track which device is which yourself and after a restart, manually assemble it. But for a short-term situation, that isn't a big problem.

An external bitmap means that if the link goes down, it keeps track of which blocks are in sync and which aren't, and when the link comes back up you re-add the missing device and the rebuild continues where it left off.

A write-mostly device means that it will never read over the slow link (unless the local device dies). It will only write over the slow link.

A write-behind device means that writes to the array will be as fast as writes to the local device. There are limits to this. Long-term, the write throughput will be limited to the slowest device, but bursts to the array will happen at the full speed of the local device.

How do you do this? First make sure you have a recent kernel and mdadm.

Then you build the array with

  mdadm --build /dev/md22 --level=1 --bitmap=/root/mybitmap \
     --write-behind --raid-disks=2 /dev/localdevice --write-mostly /dev/remotedevice

then use /dev/md22 instead of /dev/localdevice.
This will not destroy the contents of /dev/localdevice. They will remain unchanged. But it will start copying those contents to /dev/remotedevice

Obviously you need to unmount /dev/localdevice first, then set up the raid1, then mount /dev/md22 in it's place.

You will want to test this of course - I cannot guarantee that I have that command exactly right.

If the remote device fails (link breaks) it should get marked faulty in the array. Once you have it working again you can use mdadm to remove (--remove) and re-add (--re-add) the device. It should
complete the copy correctly. You should test that this actually
works as expected.

Feel free to ask questions if you choose to attempt this route.

Good luck.

NeilBrown

#10 lore raw

Neil Brown schrieb:

(...)

Quoted 10 lines

You will want to test this of course - I cannot guarantee that I have that command exactly right.

If the remote device fails (link breaks) it should get marked faulty in the array. Once you have it working again you can use mdadm to remove (--remove) and re-add (--re-add) the device. It should
complete the copy correctly. You should test that this actually
works as expected.

Feel free to ask questions if you choose to attempt this route.

Interesting lecture, thanks a lot. I'm gonna take that path.

I have one more question, though.

I want to migrate:
- from 4x400GB HDD RAID-10
- to 4x400GB HDD RAID-5

Obviously, I need 8 disks for that, but I have only 6.

So my idea was to kick one disk from RAID-10, and use it for RAID-5 for the time of migration.

This means, I would have only 3 disks out of 4 in RAID-5 array.

Is it possible to create a degraded, 4 disk RAID-5 array with just 3 drives? I would add the 4th drive once migration from RAID-10 is done.

(I'm aware of the risks - that my degraded RAID-10 will be "vulnerable" during the migration).

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#11 lore raw
Quoted 2 lines · On Tuesday May 15, [email protected] wrote:

Interesting lecture, thanks a lot. I'm gonna take that path.

:-)

Quoted 7 lines

I have one more question, though.

I want to migrate:
- from 4x400GB HDD RAID-10
- to 4x400GB HDD RAID-5

Quoted 6 lines

Obviously, I need 8 disks for that, but I have only 6.

So my idea was to kick one disk from RAID-10, and use it for RAID-5 for the time of migration.

This means, I would have only 3 disks out of 4 in RAID-5 array.

Yes... that would work.... How much do you trust your drives, and how much to do value your data? Presumably

perceived likely hood of failure * value of data < price of 2 drives.

Quoted 4 lines

Is it possible to create a degraded, 4 disk RAID-5 array with just 3 drives? I would add the 4th drive once migration from RAID-10 is done.

Certainly. When you create the raid5, use the word "missing" in place of the drive which is ... missing. This will create a degraded array. You the drive becomes available, simply add it in to the array.
If you drives are not identical (same number of sectors), make sure the last drive you add is not the smallest, else you won't be able to add it.

NeilBrown

#12 lore raw
In reply to Neil Brown

Neil Brown schrieb:

(...)

Quoted 7 lines

An external bitmap means that if the link goes down, it keeps track of which blocks are in sync and which aren't, and when the link comes back up you re-add the missing device and the rebuild continues where it left off.

  mdadm --build /dev/md22 --level=1 --bitmap=/root/mybitmap \
     --write-behind --raid-disks=2 /dev/localdevice --write-mostly /dev/remotedevice

One more question - is there a way to estimate the size of the bitmap file? Does it depend on the size of the array?

What bitmap file size can I expect for a 600 GB array?

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#13 lore raw
Quoted 9 lines · On Tuesday May 15, [email protected] wrote:

Neil Brown schrieb:

(...)

Quoted 4 lines

An external bitmap means that if the link goes down, it keeps track of which blocks are in sync and which aren't, and when the link comes back up you re-add the missing device and the rebuild continues where it left off.

Quoted 7 lines
Quoted 2 lines
  mdadm --build /dev/md22 --level=1 --bitmap=/root/mybitmap \
     --write-behind --raid-disks=2 /dev/localdevice --write-mostly /dev/remotedevice

One more question - is there a way to estimate the size of the bitmap file? Does it depend on the size of the array?

What bitmap file size can I expect for a 600 GB array?

Due to internal implementation details, mdadm limits the size of the bitmap to 2^20 bits. So the file will be 100K +/- 50%. That will lead to about 1Meg per bit. If you have a failure, this might mean you end up re-copying several megabytes more than you really need to, but that should add up to less than one second.

NeilBrown

#14 lore raw
Quoted 13 lines · Neil Brown
Quoted 12 lines · On Tuesday May 15, [email protected] wrote:

Neil Brown schrieb:

(...)

Quoted 6 lines

An external bitmap means that if the link goes down, it keeps track of which blocks are in sync and which aren't, and when the link comes back up you re-add the missing device and the rebuild continues where it left off.

  mdadm --build /dev/md22 --level=1 --bitmap=/root/mybitmap \
     --write-behind --raid-disks=2 /dev/localdevice --write-mostly /dev/remotedevice

One more question - is there a way to estimate the size of the bitmap file? Does it depend on the size of the array?

Quoted 7 lines
Quoted 1 line

What bitmap file size can I expect for a 600 GB array?

Due to internal implementation details, mdadm limits the size of the bitmap to 2^20 bits. So the file will be 100K +/- 50%. That will lead to about 1Meg per bit. If you have a failure, this might mean you end up re-copying several megabytes more than you really need to, but that should add up to less than one second.

Good, I was wondering if ~200 MB left I have on a filesystem would be enough :)

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org