Real Time Mirroring of a NAS
Hi,
I'm looking for a way to create a real-time mirror of a NAS. In other words, say I have a 5.5 TB NAS (3ware 16-drive array, RAID-5, 500 GB drives). I want to mirror it in real time to a completely separate 5.5 TB NAS. RSYNCing in the background is not an option. The two NAS boxes need to hold identical data at all times. It is NOT necessary that the data be accessed from both NAS boxes simultaneously. One is simply a backup of the other.
I guess one option might be to use Redhat's Global File System (GFS). Although there seems to be an 8TB limit for a GFS filesystem, which could be an issue down the road.
But would the following approach work:
Can I export NAS B as a SAN or ISCSI target, connect the two machines with, say, mryinet cards or 10 GbE TOE cards, mount the NAS B volume on NAS A, and create a RAID-1 mirror of the two volumes? Is this kind of thing done?
Andy Liebman
On Fri, 7 Apr 2006 at 6:47am, andy liebman wrote
Quoted 3 lines
I'm looking for a way to create a real-time mirror of a NAS. In other words, say I have a 5.5 TB NAS (3ware 16-drive array, RAID-5, 500 GB drives). I want to mirror it in real time to a completely separate 5.5 TB NAS. RSYNCing in the background is not an option. The two NAS boxes need to hold identical data at all times. It is NOT necessary that the data be accessed from both NAS boxes simultaneously. One is simply a backup of the other.
I guess one option might be to use Redhat's Global File System (GFS). Although there seems to be an 8TB limit for a GFS filesystem, which could be an issue down the road.
I don't have any experience with it, but I've often seen DRBD mentioned for just this sort of situation. I'd look into that.
Signature · 3 lines
Joshua Baker-LePain Department of Biomedical Engineering Duke University
Quoted 10 lines
Quoted 7 lines
I'm looking for a way to create a real-time mirror of a NAS. In other words, say I have a 5.5 TB NAS (3ware 16-drive array, RAID-5, 500 GB drives). I want to mirror it in real time to a completely separate 5.5 TB NAS. RSYNCing in the background is not an option. The two NAS boxes need to hold identical data at all times. It is NOT necessary that the data be accessed from both NAS boxes simultaneously. One is simply a backup of the other.
I don't have any experience with it, but I've often seen DRBD mentioned for just this sort of situation. I'd look into that.
Linux NBD (and md on top) is a simpler solution for the same thing. I'd look into that also :)
andy> I'm looking for a way to create a real-time mirror of a NAS. In andy> other words, say I have a 5.5 TB NAS (3ware 16-drive array, andy> RAID-5, 500 GB drives). I want to mirror it in real time to a andy> completely separate 5.5 TB NAS. RSYNCing in the background is andy> not an option. The two NAS boxes need to hold identical data at andy> all times. It is NOT necessary that the data be accessed from andy> both NAS boxes simultaneously. One is simply a backup of the andy> other.
How much data loss can you stand? Or more accurately, how much lag in copying data between the arrays can you stand for your Disaster recovery issues? How far apart are the two systems? Is the setup write intensive, or read intensive?
Why is rsync not an option? Not upto date enough? *grin*
One option might be to use inotify to watch filesystem A and to copy data that changes to filesystem B as quickly as possible.
andy> I guess one option might be to use Redhat's Global File System andy> (GFS). Although there seems to be an 8TB limit for a GFS andy> filesystem, which could be an issue down the road.
How large are you planning on growing your filesystem? Why can't it be split? What are your assumptions?
andy> But would the following approach work:
andy> Can I export NAS B as a SAN or ISCSI target, connect the two andy> machines with, say, mryinet cards or 10 GbE TOE cards, mount the andy> NAS B volume on NAS A, and create a RAID-1 mirror of the two andy> volumes? Is this kind of thing done?
I guess I'd look into mirroring the underlying disk subsystems and then putting the NAS on top of that. If the NAS box goes down, you can then just fire up NAS on the backup server and continue.
Mostly, I don't understand what problem you're trying to solve, so I can't offer good suggestions because the solution changes depending on assumptions.
I already assume you don't have tons of money to spend here. *grin*
John
Hello andy,
al> Can I export NAS B as a SAN or ISCSI target, connect the two machines Am I right in the assumption that your NASes are Linux boxes? :)
Did you take a look at Linux "Network block devices" (nbd/enbd)? They might be what you need: you'd get a raw device on one of the servers to use in a mirror along with a local device. The NBD page mentioned some setups for high-availability services where an active server clones itself to a backup server and vice-versa, whichever was active most recently.
I'm not sure about performance though...
al> with, say, mryinet cards or 10 GbE TOE cards, mount the NAS B volume on al> NAS A, and create a RAID-1 mirror of the two volumes? Is this kind of al> thing done?
Are you sure you need 10GbE? My experience with a 10-drive 3Ware 8506 array in RAID5 shows that reads from it usually fit in 500-700Mbit/s. And it's a very busy popular fileserver, so I guess it's close to the hardware limits of our array.
Signature · 2 lines
Best regards, Jim Klimov mailto:[email protected]
Quoted 14 lines · [email protected] wrote:
Hello andy,
al> Can I export NAS B as a SAN or ISCSI target, connect the two machines Am I right in the assumption that your NASes are Linux boxes? :)
Did you take a look at Linux "Network block devices" (nbd/enbd)? They might be what you need: you'd get a raw device on one of the servers to use in a mirror along with a local device. The NBD page mentioned some setups for high-availability services where an active server clones itself to a backup server and vice-versa, whichever was active most recently.
I'm not sure about performance though...
Quoted 9 lines
al> with, say, mryinet cards or 10 GbE TOE cards, mount the NAS B volume on al> NAS A, and create a RAID-1 mirror of the two volumes? Is this kind of al> thing done?
Are you sure you need 10GbE? My experience with a 10-drive 3Ware 8506 array in RAID5 shows that reads from it usually fit in 500-700Mbit/s. And it's a very busy popular fileserver, so I guess it's close to the hardware limits of our array.
Thanks for your reply, and the suggestions of others. I'm going to look into both NBD and DRBD.
Actually, I see that my idea to export an iSCSI target from Server B, mount it on A, and just create a RAID1 array with the two block devices must be very similar to what DRBD is doing, but my guess is that DRBD, with it's "heartbeat" signal, is probably more robust at error handling. I'd love to hear from somebody who has experience with DRBD.
By the way, I use 3ware 9550SX cards. On a 16 drive RAID-5 SATA array, I can get sequential reads that top 600 MBs/sec. That's megabytes, not megabits. And write speeds are close to 400 MB/sec with the new faster on-board XOR processing. And random reads are at least 200 MB/sec. So, 10 GbE is a must, really.
Andy
Quoted 11 lines · andy liebman
..
Thanks for your reply, and the suggestions of others. I'm going to look into both NBD and DRBD.Actually, I see that my idea to export an iSCSI target from Server B, mount it on A, and just create a RAID1 array with the two block devices must be very similar to what DRBD is doing, but my guess is that DRBD, with it's "heartbeat" signal, is probably more robust at error handling. I'd love to hear from somebody who has experience with DRBD.
Quoted 8 lines
By the way, I use 3ware 9550SX cards. On a 16 drive RAID-5 SATA array, I can get sequential reads that top 600 MBs/sec. That's megabytes, not megabits. And write speeds are close to 400 MB/sec with the new faster on-board XOR processing. And random reads are at least 200 MB/sec. So, 10 GbE is a must, really.
Andy
Hi Andy.
A couple of other suggestions that may prove helpful:
1) EVMS
http://evms.sourceforge.net/
2) Lustre
http://www.clusterfs.com/
http://www.lustre.org/
Signature · 8 lines
With our best regards, Maurice W. Hilarius Telephone: 01-780-456-9771 Hard Data Ltd. FAX: 01-780-456-9772 11060 - 166 Avenue email:[email protected] Edmonton, AB, Canada http://www.harddata.com/ T5X 1Y3
Quoted 2 lines · andy liebman
...
By the way, I use 3ware 9550SX cards. On a 16 drive RAID-5 SATA array, I can get sequential reads that top 600 MBs/sec. That's megabytes, not megabits. And write speeds are close to 400 MB/sec with the new faster on-board XOR processing. And random reads are at least 200 MB/sec. So, 10 GbE is a must, really.
A 400MB/s network input stream means either 1x10GbE or 4xGbE's to PCI-X/PCI-e bus, then to disk, then to PCI-X/PCI-e bus, then across another net to the backup system, so the I/O subsystem alone must handle 800MB/s.
3ware may actually do a single 400MB/s stream, but what about 4x100MB/s or 8x50MB/s? It's really hard to do that much I/O _and_ provide data integrity. If your data rates and the mirroring requirements are that critical you might want to look at the fast NAS systems coming out (eg- agami.com) built on top of AMD's HT architecture.
Cheers,
Quoted 7 lines
Andy
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
| for direct mail add "private_" in front of user name