Mirroring a Drive for load-balancing AND failover
I've racked my brain on this one, so hopefully someone will be of some help.
I'm trying to set up two servers which share a drive and do not have a Single Point of Failure. They are on a local network with each other.
The best solution would be to have /dev/sda1 on one server mirrored
with /dev/sda1 on the second server.
I've looked into a few solutions. One was DRBD. This would work great if I was using the second server as a hot spare, but since both server's can't have read-write access to the drbd device simultaneously, this won't work.
A second solution was to use GFS/GNBD. I can export each drive to the other server, and do RAID 1 (on both servers) between the local /dev/sda1 and the remote gnbd device. I then format the raid device with GFS so both servers can mount it.
Surprisingly, this last system works. Both systems can mount the drive and read-write to it. However, if either server in this configuration drops dead, the other server cannot deal with the dead gnbd device, and the raid device and mount point are no longer usable.
I'm sure there are numerous other problems with this setup, also.
So I'm looking for ideas. With two servers, how can I mirror a drive in real-time, and allow for failover?
Thanks,
Derek
Quoted 12 lines · Fury
I've racked my brain on this one, so hopefully someone will be of some help.
I'm trying to set up two servers which share a drive and do not have a Single Point of Failure. They are on a local network with each other.
The best solution would be to have/dev/sda1on one server mirrored
with/dev/sda1on the second server.
...
A second solution was to use GFS/GNBD. I can export each drive to the other server, and do RAID 1 (on both servers) between the local/dev/sda1and the remote gnbd device. I then format the raid device with GFS so both servers can mount it.
Quoted 8 lines
Surprisingly, this last system works. Both systems can mount the drive and read-write to it. However, if either server in this configuration drops dead, the other server cannot deal with the dead gnbd device, and the raid device and mount point are no longer usable.
I'm sure there are numerous other problems with this setup, also.So I'm looking for ideas. With two servers, how can I mirror a drive in real-time, and allow for failover?
You might want to use something more like iSCSI + RAID:
http://linux-iscsi.sourceforge.net/
--Matt
Quoted 19 lines · Matthew Gillen
Quoted 18 lines · Fury
I've racked my brain on this one, so hopefully someone will be of some help.
I'm trying to set up two servers which share a drive and do not have a Single Point of Failure. They are on a local network with each other.
The best solution would be to have/dev/sda1on one server mirrored
with/dev/sda1on the second server.
...
A second solution was to use GFS/GNBD. I can export each drive to the other server, and do RAID 1 (on both servers) between the local/dev/sda1and the remote gnbd device. I then format the raid device with GFS so both servers can mount it.Surprisingly, this last system works. Both systems can mount the drive and read-write to it. However, if either server in this configuration drops dead, the other server cannot deal with the dead gnbd device, and the raid device and mount point are no longer usable.
I'm sure there are numerous other problems with this setup, also.
Quoted 5 lines
Quoted 2 lines
So I'm looking for ideas. With two servers, how can I mirror a drive in real-time, and allow for failover?
You might want to use something more like iSCSI + RAID:
http://linux-iscsi.sourceforge.net/
How is that different than GNBD + RAID? The issue isn't the network transport, it's recovery of a RAID on two nodes simultaneously.
Signature · 8 lines
AJ Lewis Voice: 612-638-0500 Red Hat E-Mail: [email protected] One Main Street SE, Suite 209 Minneapolis, MN 55414 Current GPG fingerprint = D9F8 EDCE 4242 855F A03D 9B63 F50C 54A8 578C 8715 Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the many keyservers out there...
Quoted 25 lines · AJ Lewis
Quoted 23 lines · Matthew Gillen
Quoted 20 lines · Fury
I've racked my brain on this one, so hopefully someone will be of some help.
I'm trying to set up two servers which share a drive and do not have a Single Point of Failure. They are on a local network with each other. The best solution would be to have
/dev/sda1on one server mirrored with/dev/sda1on the second server.
...
A second solution was to use GFS/GNBD. I can export each drive to the other server, and do RAID 1 (on both servers) between the local/dev/sda1and the remote gnbd device. I then format the raid device with GFS so both servers can mount it.Surprisingly, this last system works. Both systems can mount the drive and read-write to it. However, if either server in this configuration drops dead, the other server cannot deal with the dead gnbd device, and the raid device and mount point are no longer usable. I'm sure there are numerous other problems with this setup, also.
So I'm looking for ideas. With two servers, how can I mirror a drive in real-time, and allow for failover?
Quoted 6 lines
Quoted 2 lines
You might want to use something more like iSCSI + RAID:
http://linux-iscsi.sourceforge.net/How is that different than GNBD + RAID? The issue isn't the network transport, it's recovery of a RAID on two nodes simultaneously.
I don't think he was even worried about recovery, although you're right and that's another problem. I read that he couldn't access anything after a failure of one server, which is what I was addressing.
Honestly, I don't know how GNBD works. But if it makes makes the remote volume look local and doesn't report problems in a way that RAID understands (or at all), I can see how things would hang (just like a client system would hang if an NFS server for a mounted filesystem went down). I imagine (but I don't know from personal experience) that iSCSI (with the ConnFailTimeout=x sec) would report a failed write and RAID knows how to handle that.
But, like I said, I don't know for sure about any of this, since I haven't tried it. However, the page:
http://linas.org/linux/raid.html
mentions iSCSI, so it appears that some people have gotten it to work. --Matt
I am now looking at iSCSI. The problem is exactly what Matt describes. Recovery isn't that big of a deal, when the other server comes up I can carefully set one of the devices as failed, remove it, add it again, and get them to sync. after a sync is complete on both servers, I can mount the drive on the server that went down and resume services.
Maybe iSCSI will report things diferently to md (raid). I will try.
I'm looking forward to cmirror, I hear it will be ready soon. I'm not completely sure it's what I need, maybe someone involved will see this and chime in.
-Derek
Quoted 56 lines · Matthew Gillen
Quoted 25 lines · AJ Lewis
Quoted 23 lines · Matthew Gillen
Quoted 20 lines · Fury
I've racked my brain on this one, so hopefully someone will be of some help.
I'm trying to set up two servers which share a drive and do not have a Single Point of Failure. They are on a local network with each other. The best solution would be to have
/dev/sda1on one server mirrored with/dev/sda1on the second server.
...
A second solution was to use GFS/GNBD. I can export each drive to the other server, and do RAID 1 (on both servers) between the local/dev/sda1and the remote gnbd device. I then format the raid device with GFS so both servers can mount it.Surprisingly, this last system works. Both systems can mount the drive and read-write to it. However, if either server in this configuration drops dead, the other server cannot deal with the dead gnbd device, and the raid device and mount point are no longer usable. I'm sure there are numerous other problems with this setup, also.
So I'm looking for ideas. With two servers, how can I mirror a drive in real-time, and allow for failover?
Quoted 6 lines
Quoted 2 lines
You might want to use something more like iSCSI + RAID:
http://linux-iscsi.sourceforge.net/How is that different than GNBD + RAID? The issue isn't the network transport, it's recovery of a RAID on two nodes simultaneously.
I don't think he was even worried about recovery, although you're right and that's another problem. I read that he couldn't access anything after a failure of one server, which is what I was addressing.
Honestly, I don't know how GNBD works. But if it makes makes the remote volume look local and doesn't report problems in a way that RAID understands (or at all), I can see how things would hang (just like a client system would hang if an NFS server for a mounted filesystem went down). I imagine (but I don't know from personal experience) that iSCSI (with the ConnFailTimeout=x sec) would report a failed write and RAID knows how to handle that.
But, like I said, I don't know for sure about any of this, since I haven't tried it. However, the page:
http://linas.org/linux/raid.html
mentions iSCSI, so it appears that some people have gotten it to work. --Matt_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Yes, cmirror seems like what you are looking for (along with GNBD/iSCSI).
I am testing cmirror right now, it adds the ability to:
1) have a mirrored device in a cluster environment
2) do pvmoves on clustered volumes
Many of the pieces you need are in the form of patches currently, but we're working on pulling them together... maybe a month to get the kernel changes in + user space tools cleaned up.
If anyone is interested in helping me test this beast, I'd be thankful.
I've recently uncovered a data corruption issue (I think) in cluster
mirrors when simultaneously doing looping lvs's, pvmove, and dd's from different machines. You need only a few machines and you can use gnbd for shared storage. (Actually, gnbd is the _best_ device to use for this testing, as you can forcibly remove a single device while it is in use. This obviously makes gnbd ideal for single machine mirror testing, as well.)
brassow
Quoted 79 lines · Fury
I am now looking at iSCSI. The problem is exactly what Matt describes. Recovery isn't that big of a deal, when the other server comes up I can carefully set one of the devices as failed, remove it, add it again, and get them to sync. after a sync is complete on both servers, I can mount the drive on the server that went down and resume services.
Maybe iSCSI will report things diferently to md (raid). I will try.
I'm looking forward to cmirror, I hear it will be ready soon. I'm not completely sure it's what I need, maybe someone involved will see this and chime in.
-Derek
Quoted 50 lines · Matthew Gillen
Quoted 25 lines · AJ Lewis
Quoted 23 lines · Matthew Gillen
Quoted 20 lines · Fury
I've racked my brain on this one, so hopefully someone will be of some help.
I'm trying to set up two servers which share a drive and do not have a Single Point of Failure. They are on a local network with each other. The best solution would be to have
/dev/sda1on one server mirrored with/dev/sda1on the second server.
...
A second solution was to use GFS/GNBD. I can export each drive to the other server, and do RAID 1 (on both servers) between the local/dev/sda1and the remote gnbd device. I then format the raid device with GFS so both servers can mount it.Surprisingly, this last system works. Both systems can mount the drive and read-write to it. However, if either server in this configuration drops dead, the other server cannot deal with the dead gnbd device, and the raid device and mount point are no longer usable. I'm sure there are numerous other problems with this setup, also.
So I'm looking for ideas. With two servers, how can I mirror a drive in real-time, and allow for failover?
Quoted 6 lines
Quoted 2 lines
You might want to use something more like iSCSI + RAID:
http://linux-iscsi.sourceforge.net/How is that different than GNBD + RAID? The issue isn't the network transport, it's recovery of a RAID on two nodes simultaneously.
I don't think he was even worried about recovery, although you're right and that's another problem. I read that he couldn't access anything after a failure of one server, which is what I was addressing.
Honestly, I don't know how GNBD works. But if it makes makes the remote volume look local and doesn't report problems in a way that RAID understands (or at all), I can see how things would hang (just like a client system would hang if an NFS server for a mounted filesystem went down). I imagine (but I don't know from personal experience) that iSCSI (with the ConnFailTimeout=x sec) would report a failed write and RAID knows how to handle that.
But, like I said, I don't know for sure about any of this, since I haven't tried it. However, the page:
http://linas.org/linux/raid.html
mentions iSCSI, so it appears that some people have gotten it to work. --MattQuoted 6 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/