Skip to contentSkip to conversation

Problems with software RAID + iSCSI or GNBD

DISCUSSION Christopher Smith 6 participants 9 messages Jun 27, 2005 – Jun 30, 2005 MD RAID

lore
Share & tools
lore
#0 lore raw

I'm not sure if this is the correct list to be posting this to, but it is software RAID related, so if nothing else hopefully someone here can point me in the right direction.

I'm trying to roll my own SAN, but I've had mixed results thus far. In my basic, initial setup I've created a configuration with two "disk nodes" and a single "concentrator node". My objective is to have the "concentrator" take the physical disk exported from the "disk nodes" and
stitch it together into a RAID1. So, it looks like this:

             "Concentrator"
                /dev/md0
                 /     \
             GigE       GigE
               /         \
    "Disk node 1"       "Disk node 2"

So far I've tried using iSCSI and GNBD as the "back end" to make the disk space in the nodes visible to the concentrator. I've had two problems, one unique to using iSCSI and the other common to both.

Problem 1: (Re)Sync performance is atrocious with iSCSI

If I use iSCSI as the back end, the RAID only builds at about 6 - 7M/sec. Once that is complete, however, performance is much better - reads around 100M/sec and writes around 50M/sec. It's only during the sync the performance is awful. It's not related to /proc/sys/dev/raid/speed_limit_max either, which I have set to 50M/sec.
Nor is it related to the sheer volume of traffic flying around, as if
I use disktest to simultaneously read and write to both disk nodes, performance on all benchmarks only drops down to about 40 - 50M/sec.

If I switch the back end to GNBD, the resync speed is around 40 - 50M/sec.

Problem 2: The system doesn't deal with failure very well.

Once I got the RAID1 up and running, I tried to simulate a node failure by pulling the network cable from the node while disk activity was taking place. I was hoping the concentrator would detect the "disk" had failed and simply drop it from the array (so it could later be simply re-added). Unfortunately that doesn't appear to happen. What does happen is that all IO to the md device "hangs" (eg: disktest throughput drops to 0M/sec), I am unable to either 'cat /prod/mdstat' to see the md device's status or use mdadm to manually fail the device - both simply result in the command "hanging".

Does anyone have any insight as to what might be causing these problems ?

Cheers,
CS

#1 lore raw
Quoted 6 lines

Problem 2: The system doesn't deal with failure very well.

Once I got the RAID1 up and running, I tried to simulate a node failure by pulling the network cable from the node while disk activity was taking place. I was hoping the concentrator would detect the "disk" had failed and simply drop it from the array (so it could later be simply re-added). Unfortunately that doesn't appear to happen. What does happen is that all IO to the md device "hangs" (eg: disktest throughput drops to 0M/sec), I am unable to either 'cat /prod/mdstat' to see the md device's status or use mdadm to manually fail the device
- both simply result in the command "hanging".

I probably can't help much at the moment, but...

I didn't realize that NBD type things had advanced to even this level of stability. This is good news. I've been wanting to do something like what you're trying for some time to overcome the bounds of power/busses/heat/space that limit you to a single machine when building a large md or LVM. Looking at the GNBD project page, it still seems pretty raw, although a related project DDRAID seems to hold some promise.

I'd be pretty timid about putting anything close to production on these drivers, though.

What distro / kernel version / level of GNBD are you using?

Regards-
Michael Stumpf

#3 lore raw
In reply to Christopher Smith
Quoted 10 lines · Christopher Smith

stitch it together into a RAID1. So, it looks like this:

             "Concentrator"
                /dev/md0
                 /     \
             GigE       GigE
               /         \
    "Disk node 1"       "Disk node 2"

So far I've tried using iSCSI and GNBD as the "back end" to make the disk space in the nodes visible to the concentrator. I've had two problems, one unique to using iSCSI and the other common to both.

Personally, I wouldn't mess with iSCSI or GNBD. You don't need GNBD in this scenario anyway; simple nbd (which is in the mainline kernel...get the userland tools at sourceforge.net/projects/nbd) will do just fine, and I'd be willing to bet that it is more stable and faster...

Quoted 3 lines

Problem 2: The system doesn't deal with failure very well.

Once I got the RAID1 up and running, I tried to simulate a node failure by pulling the network cable from the node while disk activity was taking place. I was hoping the concentrator would detect the "disk" had failed and simply drop it from the array (so it could later be simply re-added). Unfortunately that doesn't appear to happen. What does happen is that all IO to the md device "hangs" (eg: disktest throughput drops to 0M/sec), I am unable to either 'cat /prod/mdstat' to see the md device's status or use mdadm to manually fail the device - both simply result in the command "hanging".

Well, that's the fault of either iSCSI or GNBD. md/raid1 over nbd works flawlessly in this scenario on 2.6 kernels (for 2.4, you'll need a special patch -- ask me for it, if you want).

--
Paul

#2 lore raw
In reply to Michael Stumpf
Quoted 7 lines · Michael Stumpf

I probably can't help much at the moment, but...

I didn't realize that NBD type things had advanced to even this level of stability. This is good news. I've been wanting to do something like what you're trying for some time to overcome the bounds of power/busses/heat/space that limit you to a single machine when building a large md or LVM. Looking at the GNBD project page, it still seems pretty raw, although a related project DDRAID seems to hold some promise.

I'd be pretty timid about putting anything close to production on these drivers, though.

What distro / kernel version / level of GNBD are you using?

Well, I don't know if they have yet - the main reason I'm fiddling with this is to see if it's feasible :).

However, I have belted tens of gigabytes of data at the mirror-over-GNBD and mirror-over-iSCSI using various benchmarking tools without any kernel panics or (apparent) data corruption, so I'm gaining confidence that it's a workable solution. I haven't yet started the same level of testing with Windows and Linux clients sitting above the initiator/bridge level yet, however, as I want to make sure the back end is pretty stable before moving on (as it will become a - relatively - single point of failure for most of the important machines in our network, and hence the entire company).

I'm just using a stock Fedora Core 4 and the GNBD it includes. A bit bleeding edge, I know, but I figured since it had just been released when I started on this project, why not ;).

With regards to the problem I was having with node failures, at least with iSCSI the solution was setting a timeout so that a "disk failed" error was actually returned - by default the iSCSI initiator assumes any disconnection errors are network-related and transient, so it simply stops any IO to the iSCSI target until it reappears. Now that I've specified a timeout, node "failures" behave as expected and the mirror goes into degraded mode.

I assume I need to do something similar with GNBD so that it really does "fail", rather than "hang", but I've been too busy over the last few days to actually look into it.

CS

#4 lore raw
In reply to Paul Clements
Quoted 11 lines · Paul Clements
Quoted 9 lines · Christopher Smith

stitch it together into a RAID1. So, it looks like this:

             "Concentrator"
                /dev/md0
                 /     \
             GigE       GigE
               /         \
    "Disk node 1"       "Disk node 2"
Quoted 4 lines
Quoted 1 line

So far I've tried using iSCSI and GNBD as the "back end" to make the disk space in the nodes visible to the concentrator. I've had two problems, one unique to using iSCSI and the other common to both.

Personally, I wouldn't mess with iSCSI or GNBD. You don't need GNBD in this scenario anyway; simple nbd (which is in the mainline kernel...get the userland tools at sourceforge.net/projects/nbd) will do just fine, and I'd be willing to bet that it is more stable and faster...

I'd briefly tried nbd, but decided to look elsewhere since it needed to much manual configuration (no included rc script, /dev nodes appear to have to be manually created - yes, I'm lazy).

I've just finished trying NBD now and it seems to solve both my problems - rebuild speed is a healthy 40MB/sec + and the failures are dealt with "properly" (ie: the md device goes into degraded mode if a component nbd suddenly disappears). This looks like it might be a goer for the disk node/RAID-over-network back-end.

On the "front end", however, we have to use iSCSI because we're planning on doling the aggregate disk space out to a mix of platforms (some of them potentially clustered in the future) so they can re-share the space to "client" machines. However, the IETD iSCSI Target seems pretty solid, so I'm not really concerned about that.

Quoted 1 line

Well, that's the fault of either iSCSI or GNBD. md/raid1 over nbd works flawlessly in this scenario on 2.6 kernels (for 2.4, you'll need a special patch -- ask me for it, if you want).

Yeah, I fixed that problem (at least with iSCSI - haven't tried with GNBD). It was a PEBKAC issue :).

#5 lore raw
Quoted 8 lines · Christopher Smith
Quoted 4 lines

Personally, I wouldn't mess with iSCSI or GNBD. You don't need GNBD in this scenario anyway; simple nbd (which is in the mainline kernel...get the userland tools at sourceforge.net/projects/nbd) will do just fine, and I'd be willing to bet that it is more stable and faster...

I'd briefly tried nbd, but decided to look elsewhere since it needed to much manual configuration (no included rc script, /dev nodes appear to have to be manually created - yes, I'm lazy).

Based on one test of nbd, it seems to be stable. I did about what you are trying, a RAID1 to create an md device, then did an encrypted filesystem on it. My test was minor, throw a lot of data at it, check that it is there (md5sum), reboot and verify everything still works, drop the local drive and rebuild. I did NOT try a rebuild on the nbd drive.

Quoted 6 lines

I've just finished trying NBD now and it seems to solve both my problems
- rebuild speed is a healthy 40MB/sec + and the failures are dealt with "properly" (ie: the md device goes into degraded mode if a component nbd suddenly disappears). This looks like it might be a goer for the disk node/RAID-over-network back-end.

I failed it manually, can't say what pulling the plug will do. Glad it's working, I may be doing this on a WAN in the fall.

Signature · 1 line
bill davidsen <[email protected]>
#6 lore raw

This discussion intrigues me. I think there is a lot of merit to running a raid in this manner. However, if I understand correctly, under normal circumstances reads from a Raid1 md device will always round-robin between the devices to increase performance. Is there a way or what would need to be done to set a single component in the device to be the primary. ie. don't read from the other device unless the first one fails.
I think there was some discussion about this a month or so ago concerning ramdisk(which I don't know would be quite as useful), but the theory can apply to any block devices with significantly different speed/latencies, etc.
Please advise.
--David Dougall

Quoted 37 lines · Bill Davidsen
Quoted 8 lines · Christopher Smith
Quoted 4 lines

Personally, I wouldn't mess with iSCSI or GNBD. You don't need GNBD in this scenario anyway; simple nbd (which is in the mainline kernel...get the userland tools at sourceforge.net/projects/nbd) will do just fine, and I'd be willing to bet that it is more stable and faster...

I'd briefly tried nbd, but decided to look elsewhere since it needed to much manual configuration (no included rc script, /dev nodes appear to have to be manually created - yes, I'm lazy).

Based on one test of nbd, it seems to be stable. I did about what you are trying, a RAID1 to create an md device, then did an encrypted filesystem on it. My test was minor, throw a lot of data at it, check that it is there (md5sum), reboot and verify everything still works, drop the local drive and rebuild. I did NOT try a rebuild on the nbd drive.

Quoted 6 lines

I've just finished trying NBD now and it seems to solve both my problems
- rebuild speed is a healthy 40MB/sec + and the failures are dealt with "properly" (ie: the md device goes into degraded mode if a component nbd suddenly disappears). This looks like it might be a goer for the disk node/RAID-over-network back-end.

I failed it manually, can't say what pulling the plug will do. Glad it's working, I may be doing this on a WAN in the fall.

--
bill davidsen <[email protected]>

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 4 lines · David Dougall

This discussion intrigues me. I think there is a lot of merit to running a raid in this manner. However, if I understand correctly, under normal circumstances reads from a Raid1 md device will always round-robin between the devices to increase performance. Is there a way or what would need to

Quoted 8 lines

be done to set a single component in the device to be the primary. ie. don't read from the other device unless the first one fails. I think there was some discussion about this a month or so ago concerning ramdisk(which I don't know would be quite as useful), but the theory can apply to any block devices with significantly different speed/latencies, etc.
Please advise.
--David Dougall

It would be useful to prioritize device selection in mirrored arrays, such that for reads one device would be used until the i/o queue had N entries, then start using the next. It would even seem that the prioritization could be set by the kernel, "by observation of seek and transfer times." In the case of a connection via GigE I doubt the network time is an issue compared to the physical processes. If the link is T1 between continents it's an issue for big data, and if one end is connected via PPP over dialup (looking at the worst case) you probably don't want to read over the net unless the local device has flames coming out.

Other common cases like slow disk on a fast bus and a fast disk on a slower bus might really mess up any auto tuning. The common case is a laptop with a 5400 rpm PATA drive internal and a 7200 rpm fast seek drive plugged into USB2.

I don't see a huge benefit unless there is a really slow network involved. Perhaps some developer could speak to the possibility of preferential reads in general. I peeked at the code, and while it looks actually fairly simple, it also feels as if it might have a lot of "unintended consequences," and violations of Plauger's "Law of Least Astonishment."

Signature · 1 line
bill davidsen <[email protected]>
#8 lore raw
In reply to David Dougall
Quoted 3 lines · David Dougall

reads from a Raid1 md device will always round-robin between the devices to increase performance.

If it was round-robin that would actually decrease the performance, it's locality based. It reads from the RAID member that last operated in the physical vicinity of the requested data to be read. ie it minimize the distance disk heads have to move. This is one way RAID1 lowers average seek time.

-ryan