Skip to contentSkip to conversation

md with shared disks

DISCUSSION Anton Ekermans 3 participants 7 messages Nov 9, 2014 – Nov 14, 2014 MD RAID

lore
Share & tools
lore
#0 lore raw

Good day raiders,
I have a question on md that I cannot find (up to date) answer to.
We use SuperMicro server with 16 shared disks on a shared backplane between two motherboards, running up to date CentOS7.
If I create an array on one node, the other node can detect it. I put GFS2 on top of the array so both system can share the filesystem, but I want to know if md raid is safe to be used in this way with possibly 2 active/active nodes changing the metadata at the same time. I've disabled raid-check cron job on one node so they don't both resync the drives weekly, but I suspect there's a lot more to it than that.

If it's not possible, then alternatively some advice on strategy to have a large active/active shared disk/filesystem would also be welcome.

Best regards

Untitled Document

Anton Ekermans
Technical/R&D
E-mail: antone true co za
Tel: 042 293 4168 Fax: 042 293 1851
Web: www.true.co.za <http://www.true.co.za>

#1 lore raw
Quoted 6 lines · Anton Ekermans

Good day raiders,
I have a question on md that I cannot find (up to date) answer to.
We use SuperMicro server with 16 shared disks on a shared backplane between two motherboards, running up to date CentOS7.
If I create an array on one node, the other node can detect it. I put GFS2 on top of the array so both system can share the filesystem, but I want to know if md raid is safe to be used in this way with possibly 2 active/active nodes changing the metadata at the same time. I've disabled raid-check cron job on one node so they don't both resync the drives weekly, but I suspect there's a lot more to it than that.

If it's not possible, then alternatively some advice on strategy to have a large active/active shared disk/filesystem would also be welcome.

Not possible, as far as I know: MD does not reload / exchange metadata information with other MD peers. MD thinks it is the only user of those disks.
If you attempt to share the arrays and then one head fails one disk and starts reconstruction onto another disk, while the other head thinks the array is all right, havoc will arise certainly.

Even without this worst-case scenario, data probably will be still lost because the two MDs are not cache coherent, so writes on one head will not invalidate the kernel cache for the same region on the other head, and this is bad because reads performed on the other head will not see the changes just written if such area was cached in the kernel. GFS actually will attempt to invalidate such cache but I am not sure to what extent: if you use raid5/6 probably it is not enough because the stripe-cache will hold stale data in a way that GFS probably does not know about (does not go away even with echo 3 > /proc/sys/vm/drop_caches ). Maybe raid0/1/10 can be safer... anybody knows if cache dropping works well there? But the problem of consistent vision of disk failures and raid reconstruction seems harder to overcome.

You can do an active/passive configuration, shutting down MD on one head and starting it on the other head.
Another option is the crossed-active or whatever it is called: some arrays are active on one head node, other arrays on the other head node, so to share the computational and bandwidth burden.

If other people have better ideas I am all ears.

Regards
EW

#2 lore raw
In reply to Anton Ekermans
Quoted 11 lines · Anton Ekermans

Good day raiders,
I have a question on md that I cannot find (up to date) answer to. We use SuperMicro server with 16 shared disks on a shared backplane between two motherboards, running up to date CentOS7.
If I create an array on one node, the other node can detect it. I put GFS2 on top of the array so both system can share the filesystem, but I want to know if md raid is safe to be used in this way with possibly 2 active/active nodes changing the metadata at the same time. I've disabled raid-check cron job on one node so they don't both resync the drives weekly, but I suspect there's a lot more to it than that.

Quoted 2 lines

If it's not possible, then alternatively some advice on strategy to have a large active/active shared disk/filesystem would also be welcome.

It's not possible to do what you mention as md is not cluster aware. It will break, badly. What most people do in such cases in create two md arrays, one controlled by each host, and mirror them with DRBD, then put OCFS/GFS atop DRBD. You lose half your capacity doing this, but it's the only way to do it and have all disks active. Of course you lose half your bandwidth as well. This is a high availability solution, not high performance.

You bought this hardware to do something. And that something wasn't simply making two hosts in one box use all the disks in the box. What is the workload you plan to run on this hardware? The workload dictates the needed hardware architecture, not the other way around. If you want high availability this hardware will work using the stack architecture above, and work well. If you need high performance shared filesystem access between both nodes you need an external SAS/FC RAID array and a cluster FS. In either case you're using a cluster FS which means high file throughput but low metadata throughgput.

If it's high performance you need, an option is to submit patches to make md cluster aware. Another is the LSI clustering RAID controller kit for internal drives. Don't know anything about it other than it is available and apparently works with RHEL and SUSE. Seems suitable for what you express as your need.

http://www.lsi.com/products/shared-das/pages/syncro-cs-9271-8i.aspx#tab/tab2

Cheers,
Stan

#3 lore raw

Thank you very much for your clear response.
The purpose of this hardware is to primarily host ample VM storage for the 2 nodes itself and 3 other i7 PC/servers.
The HA was hoped to be achieved as active/active with both nodes sharing the same disks and non-cluster servers(i7) having multi-path to these two nodes. This is advertised as HA active/active in storage software such as Nexenta using RSF-1. However upon closer inspection, their active/active means both nodes share some data and the other can take over. So for me, in essence it is "active/passive + passive/active" and not truly "active/active". We will try to config this way to get quasi active/active for best performance with kind-of high-availability. Seems the shared disks is not the problem, but combining them on a cluster is.

Thank you again

Best regards
Untitled Document

Anton Ekermans

Quoted 29 lines

It's not possible to do what you mention as md is not cluster aware. It will break, badly. What most people do in such cases in create two md arrays, one controlled by each host, and mirror them with DRBD, then put OCFS/GFS atop DRBD. You lose half your capacity doing this, but it's the only way to do it and have all disks active. Of course you lose half your bandwidth as well. This is a high availability solution, not high performance.

You bought this hardware to do something. And that something wasn't simply making two hosts in one box use all the disks in the box. What is the workload you plan to run on this hardware? The workload dictates the needed hardware architecture, not the other way around. If you want high availability this hardware will work using the stack architecture above, and work well. If you need high performance shared filesystem access between both nodes you need an external SAS/FC RAID array and a cluster FS. In either case you're using a cluster FS which means high file throughput but low metadata throughgput.

If it's high performance you need, an option is to submit patches to make md cluster aware. Another is the LSI clustering RAID controller kit for internal drives. Don't know anything about it other than it is available and apparently works with RHEL and SUSE. Seems suitable for what you express as your need.

http://www.lsi.com/products/shared-das/pages/syncro-cs-9271-8i.aspx#tab/tab2

Cheers,
Stan

#4 lore raw

With DRBD and GFS2 it is true active/active at the block level. You just lose half your disk capacity due to the host-to-host mirroring. Whether your upper layers are active/active is another story. E.g. getting NFS server/client to do seamless automatic path failover is still a shaky proposition AIUI.

You mention multipath. If you plan to use iSCSI multipath for the i7 servers you need to make sure each LUN you export has the same WWID on both cluster nodes.

Stan

Quoted 51 lines · Anton Ekermans

Thank you very much for your clear response.
The purpose of this hardware is to primarily host ample VM storage for the 2 nodes itself and 3 other i7 PC/servers.
The HA was hoped to be achieved as active/active with both nodes sharing the same disks and non-cluster servers(i7) having multi-path to these two nodes. This is advertised as HA active/active in storage software such as Nexenta using RSF-1. However upon closer inspection, their active/active means both nodes share some data and the other can take over. So for me, in essence it is "active/passive + passive/active" and not truly "active/active". We will try to config this way to get quasi active/active for best performance with kind-of high-availability. Seems the shared disks is not the problem, but combining them on a cluster is.

Thank you again

Best regards
Untitled Document

Anton Ekermans

Quoted 30 lines

It's not possible to do what you mention as md is not cluster aware. It will break, badly. What most people do in such cases in create two md arrays, one controlled by each host, and mirror them with DRBD, then put OCFS/GFS atop DRBD. You lose half your capacity doing this, but it's the only way to do it and have all disks active. Of course you lose half your bandwidth as well. This is a high availability solution, not high performance.

You bought this hardware to do something. And that something wasn't simply making two hosts in one box use all the disks in the box. What is the workload you plan to run on this hardware? The workload dictates the needed hardware architecture, not the other way around. If you want high availability this hardware will work using the stack architecture above, and work well. If you need high performance shared filesystem access between both nodes you need an external SAS/FC RAID array and a cluster FS. In either case you're using a cluster FS which means high file throughput but low metadata throughgput.

If it's high performance you need, an option is to submit patches to make md cluster aware. Another is the LSI clustering RAID controller kit for internal drives. Don't know anything about it other than it is available and apparently works with RHEL and SUSE. Seems suitable for what you express as your need.

http://www.lsi.com/products/shared-das/pages/syncro-cs-9271-8i.aspx#tab/tab2

Cheers,
Stan

#5 lore raw
Quoted 2 lines · Stan Hoeppner

With DRBD and GFS2 it is true active/active at the block level. You just lose half your disk capacity due to the host-to-host mirroring.

Sorry but I don't share your definition of active/active.

Would you say that a raid1 is an active/active thing?

Doubling the number of disks and repeating the operation on both sides is not active/active in the sense that people usually want.

Active/active commonly means that you have twice the performance of active/passive.

In this sense DRBD not only is an active/passive but it is even way below the performances of an active/passive because it has to transmit the data to the peer in addition to write to the disks, and this takes CPU time for memcpy and interrupts, introduces latency, requires additional hardware (= fast networking dedicated to DRBD). An active/passive with shared disks is hence "twice" (very roughly) faster than DRBD at the same price spent on the head nodes. An active/active with shared disks is hence 4 times (again very roughly) faster than DRBD, at the same price for the head nodes.

In addition to this with DRBD you have to buy twice the number of disks, which is also an additional expense. Marginally though, because a shared-disk infrastructure is way more expensive than a direct-attached one, but it has to be planned like that in advance, and not retrofitted like you propose.

His current infrastructure cannot be easily converted to DRBD without major losses: if he attempts to do so he will have almost double the costs of a basic DRBD shared-nothing direct-attached infrastructure or exactly double the cost of a shared-disk infrastructure, intended as cost per TB of data. Unfortunately, after this he will still have half the performances of an active/passive shared-disk clustered-MD solution.

#6 lore raw
Quoted 30 lines · Ethan Wilson
Quoted 2 lines · Stan Hoeppner

With DRBD and GFS2 it is true active/active at the block level. You just lose half your disk capacity due to the host-to-host mirroring.

Sorry but I don't share your definition of active/active.

Would you say that a raid1 is an active/active thing?

Doubling the number of disks and repeating the operation on both sides is not active/active in the sense that people usually want.

Active/active commonly means that you have twice the performance of active/passive.

In this sense DRBD not only is an active/passive but it is even way below the performances of an active/passive because it has to transmit the data to the peer in addition to write to the disks, and this takes CPU time for memcpy and interrupts, introduces latency, requires additional hardware (= fast networking dedicated to DRBD). An active/passive with shared disks is hence "twice" (very roughly) faster than DRBD at the same price spent on the head nodes. An active/active with shared disks is hence 4 times (again very roughly) faster than DRBD, at the same price for the head nodes.

In addition to this with DRBD you have to buy twice the number of disks, which is also an additional expense. Marginally though, because a shared-disk infrastructure is way more expensive than a direct-attached one, but it has to be planned like that in advance, and not retrofitted like you propose.

Quoted 6 lines

His current infrastructure cannot be easily converted to DRBD without major losses: if he attempts to do so he will have almost double the costs of a basic DRBD shared-nothing direct-attached infrastructure or exactly double the cost of a shared-disk infrastructure, intended as cost per TB of data. Unfortunately, after this he will still have half the performances of an active/passive shared-disk clustered-MD solution.

He doesn't have an infrastructure yet. He's attempting to build one but purchased the wrong gear for his requirements. I presented him with options to do it the right way, and to salvage what he has already purchased. The DRBD active/active option is the latter. The SAN option was the former. You seem to have misunderstood my comments.

Cheers,
Stan