Linux SATA RAID FAQ
As author of the current Linux SATA driver, I get the brunt of the questions and "bug reports" about "Linux doesn't support my hardware SATA RAID". Sigh. Silly marketing departments.
Thus, I have created a FAQ. You might sense a theme...
http://www.kernel.org/pub/linux/kernel/people/jgarzik/faq-sata-raid.html
Quoted 3 lines
Thus, I have created a FAQ. You might sense a theme...
http://www.kernel.org/pub/linux/kernel/people/jgarzik/faq-sata-raid.html
;-) Funny.....
I don't care about RAID, but could you tell which S-ATA Controller/Chipset is the best for Hot-Swap?
Thanks,
Michael
Quoted 2 lines · Michael Knigge
I don't care about RAID, but could you tell which S-ATA Controller/Chipset is the best for Hot-Swap?
Hi Michael,
I think 3ware is the most stable right now, and has his own separate driver. It is even seen as a SCSI controller by linux, but has SATA or PATA disks.
Maybe there are some others, but I heard this is stable right now, and hot-swapping is working on it.
Hi Jeff,
I like it. It's fairly simple. I'm always amazed how many people do really believe that these cards provide hardware RAID !!! The problem is when you ask a reseller to add a real hardware RAID card in a system you purchase and you end up with a cheap silicon image... It happened to us once and it's not funny at all.
Cheers,
Willy
Quoted 5 lines · In article <[email protected]> you wrote:
I like it. It's fairly simple. I'm always amazed how many people do really believe that these cards provide hardware RAID !!! The problem is when you ask a reseller to add a real hardware RAID card in a system you purchase and you end up with a cheap silicon image... It happened to us once and it's not funny at all.
Speaing of that, does that mean that other OSes (i.e. Windows) are using BIOS provided code to do raid, or do they also have raid software drivers and the bios is only used on bootup for signature detection and formatting?
Greetings
Bernd
Signature · 2 lines
eckes privat - http://www.eckes.org/ Project Freefire - http://www.freefire.org/
Quoted 3 lines · Bernd Eckenfels
Speaing of that, does that mean that other OSes (i.e. Windows) are using BIOS provided code to do raid, or do they also have raid software drivers and the bios is only used on bootup for signature detection and formatting?
Normally BIOS and windows drivers doing their raid. It isn't entirely that simple. The 3ware is hardware raid as are some of the other high end devices (eg aacraid sata boards). There are also some low end devices with part of the raid logic in hardware (some promise) although I don't believe we use that to the full yet.
I'm currently trying to fix up the IT8212 which is an older PATA board which does have real h/w raid 0/1
Alan
Quoted 3 lines · Alan Cox
end devices (eg aacraid sata boards). There are also some low end devices with part of the raid logic in hardware (some promise) although I don't believe we use that to the full yet.
Nope. My SATA RAID FAQ mentions the Promise "RAID accelerator" stuff. In particular, the Promise SX4 is in fact likely to be faster with the _Promise_ RAID driver, rather than the Linux non-RAID driver.
I really like the SX4, and wish Linux could make better use of it...
The SX4 has an on-board DIMM (128M - 2G), through which all data _must_ pass. The data transfer between host and on-board DIMM is a separate DMA engine and separate interrupt event from the four ATA DMA engines (one per SATA port). There are several possibilities that are worth exploring on this card:
* Caching
* Eliminate PCI bus traffic by sending RAID1/5 writes a _single_ time to the card, and then multiplex to multiple attached drives from there
* Offload RAID5 XOR calculations, which becomes quite useful in combination with these other features
* Execute RAID1/5 resyncs and parity checks completely on the card
I have full docs and knowledge, but no idea how to hook this stuff into Linux md efficiently.
Jeff
Quoted 8 lines · Jeff Garzik
Quoted 3 lines · Alan Cox
end devices (eg aacraid sata boards). There are also some low end devices with part of the raid logic in hardware (some promise) although I don't believe we use that to the full yet.
Nope. My SATA RAID FAQ mentions the Promise "RAID accelerator" stuff.
Clarification: "nope" == "nope, we don't use that to the full yet"
Quoted 6 lines
The SX4 has an on-board DIMM (128M - 2G), through which all data _must_ pass. The data transfer between host and on-board DIMM is a separate DMA engine and separate interrupt event from the four ATA DMA engines (one per SATA port). There are several possibilities that are worth exploring on this card:
* Caching
* Eliminate PCI bus traffic by sending RAID1/5 writes a _single_ time to the card, and then multiplex to multiple attached drives from there
* Offload RAID5 XOR calculations, which becomes quite useful in combination with these other features
* Execute RAID1/5 resyncs and parity checks completely on the card
And one more: the Promise hardware allows multiple disk transactions to be chained together in a sequence, such that, you only receive an interrupt when the full sequence is complete (or there is an error). You can look at it as either interrupt coalescing, or simply coalescing of multiple low-level disk transactions into a single "RAID transaction."
Jeff
Quoted 7 lines · Jeff Garzik
Quoted 7 lines
The SX4 has an on-board DIMM (128M - 2G), through which all data _must_ pass. The data transfer between host and on-board DIMM is a separate DMA engine and separate interrupt event from the four ATA DMA engines (one per SATA port). There are several possibilities that are worth exploring on this card:
* Caching
Is it battery backed ? If it is battery backed then its useful, if not then it becomes less useful although not always. The i2o drivers have some ioctls so you can turn on writeback caching even without battery backup. While this is suicidal for filesytems its just great for swap..
Quoted 7 lines · Alan Cox
Quoted 6 lines · Jeff Garzik
Quoted 6 lines
The SX4 has an on-board DIMM (128M - 2G), through which all data _must_ pass. The data transfer between host and on-board DIMM is a separate DMA engine and separate interrupt event from the four ATA DMA engines (one per SATA port). There are several possibilities that are worth exploring on this card:
Quoted 6 lines
Quoted 1 line
Quoted 1 line
* Caching
Is it battery backed ? If it is battery backed then its useful, if not then it becomes less useful although not always. The i2o drivers have some ioctls so you can turn on writeback caching even without battery backup. While this is suicidal for filesytems its just great for swap..
Nope not battery backed...
Jeff
Quoted 6 lines · Alan Cox
Quoted 1 line
Quoted 1 line
* Caching
Is it battery backed ? If it is battery backed then its useful, if not then it becomes less useful although not always. The i2o drivers have some ioctls so you can turn on writeback caching even without battery backup. While this is suicidal for filesytems its just great for swap..
Isn't sufficient to have it do ordered writes ? If you power your machine off, you'll have things half-written anyway, the only thing important with journaled filesystems (and raid5 arrays) is to have writes staying between barriers.
Xav
Quoted 8 lines · Xavier Bestel
Quoted 6 lines · Alan Cox
Quoted 1 line
Quoted 1 line
* Caching
Is it battery backed ? If it is battery backed then its useful, if not then it becomes less useful although not always. The i2o drivers have some ioctls so you can turn on writeback caching even without battery backup. While this is suicidal for filesytems its just great for swap..
Quoted 4 lines
Isn't sufficient to have it do ordered writes ? If you power your machine off, you'll have things half-written anyway, the only thing important with journaled filesystems (and raid5 arrays) is to have writes staying between barriers.
On a RAID controller with battery backed write-back cache, it can complete a "sync" operation as soon as the data is in the controller
cache. This gives a significant performance speedup.
If the cache is not battery backed and power is lost after such a "sync" operation, then data that the kernel/userspace thought was sync'ed to disk is actually lost.
This *will* break journalling filesystems and databases. They work from the assumption that 'sync' means 'sync' - and if it doesn't, then all hell breaks lose.
It is common to enforce ordering of writes by issuing a 'sync' of some data and after the sync completes then starting the writeout of the 'next' data. On a controller with write-back cache without battery backup, you could actually risk that your 'next' data were written before the data you just issued a 'sync' for.
In other words; write-back cache without battery backup is absolutely insane, except for some few isolated cases, such as swap-space that Alan pointed out.
Signature · 1 line
/ jakob
Quoted 4 lines · Xavier Bestel
Isn't sufficient to have it do ordered writes ? If you power your machine off, you'll have things half-written anyway, the only thing important with journaled filesystems (and raid5 arrays) is to have writes staying between barriers.
True to a point but 2Gb of data going walkies will offend even if the file system is gloriously intact
Quoted 2 lines · Alan Cox
True to a point but 2Gb of data going walkies will offend even if the file system is gloriously intact
Especially if the data was already visible in the shared filesystem space.
Greetings
Bernd
Signature · 4 lines
(OO) -- Bernd_Eckenfels@Mörscher_Strasse_8.76185Karlsruhe.de --
( .. ) ecki@{inka.de,linux.de,debian.org} http://www.eckes.org/
o--o 1024D/E383CD7E eckes@IRCNet v:+497211603874 f:+497211606754
(O____O) When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!Quoted 3 lines · Alan Cox
I'm currently trying to fix up the IT8212 which is an older PATA board
which does have real h/w raid 0/1
I'm sitting on the vendor's driver for these cards. How does your work differ from this?
hch questioned why we need the driver at all: just put the card in JBOD mode and use s/w raid drivers. But the thing does have an on-board CPU and the idea is that by offloading to that, the data transits the bus just a single time. The developers are off doing some comparative benchmarking at present.
Quoted 5 lines · Andrew Morton
hch questioned why we need the driver at all: just put the card in JBOD mode and use s/w raid drivers. But the thing does have an on-board CPU and the idea is that by offloading to that, the data transits the bus just a single time. The developers are off doing some comparative benchmarking at present.
Alan's driver can use the chip in raid mode.
Quoted 2 lines · Andrew Morton
I'm sitting on the vendor's driver for these cards. How does your work differ from this?
It uses the IDE layer instead of badly duplicating it in essence.
Quoted 5 lines
hch questioned why we need the driver at all: just put the card in JBOD mode and use s/w raid drivers. But the thing does have an on-board CPU and the idea is that by offloading to that, the data transits the bus just a single time. The developers are off doing some comparative benchmarking at present.
On my set up raid1 is materially faster using their processor and raid0 is materially faster not. There are also co-existance issues with Windows dual boot setups. The PCI single copy makes a big difference on a 32bit/33Mhz plug in card.
Alan
Quoted 7 lines · you
Quoted 5 lines · In article <[email protected]> you wrote:
I like it. It's fairly simple. I'm always amazed how many people do really believe that these cards provide hardware RAID !!! The problem is when you ask a reseller to add a real hardware RAID card in a system you purchase and you end up with a cheap silicon image... It happened to us once and it's not funny at all.
Quoted 6 lines
Speaing of that, does that mean that other OSes (i.e. Windows) are using BIOS provided code to do raid, or do they also have raid software drivers and the bios is only used on bootup for signature detection and formatting?
Greetings
Bernd
Windows also has "software drivers". Since vendors provide drivers independently of Microsoft, they can put precisely what they like in them.
Signature · 8 lines
Cheers,
Alistair.
personal: alistair()devzero!co!uk
university: s0348365()sms!ed!ac!uk
student: CS/AI Undergraduate
contact: 1F2 55 South Clerk Street,
Edinburgh. EH8 9PP.Quoted 6 lines · Willy Tarreau
I like it. It's fairly simple. I'm always amazed how many people do really believe that these cards provide hardware RAID !!! The problem is when you ask a reseller to add a real hardware RAID card in a system you purchase and you end up with a cheap silicon image... It happened to us once and it's not funny at all.
On the brightside, md raid5 is often faster than hardware raid5. At least on the 7000 and 8000 series of 3ware hardware; the 9000 series looks promising though. I haven't seen megaraid SATA numbers, and I don't know what happened to the SX8.
When the libata Marvell drivers come out, you'll have a cheap upgrade path for PCI-X boards if you want fast md raid: http://www.supermicro.com/products/accessories/addon/DAC-SATA-MV8.cfm $100 to add 8 unbottlenecked SATA ports to your server motherboard.
-ryan
Yup SATA is catching on quickly.
I expect to be releasing a new SATA/RAID driver here within a few weeks as well. PCI-X, TCQ, H/W RAID 0/1/10, ..
Cheers
Signature · 2 lines
Mark Lord (hdparm keeper & the original "Linux IDE Guy")
Quoted 4 lines · Mark Lord
Yup SATA is catching on quickly.
I expect to be releasing a new SATA/RAID driver here within a few weeks as well. PCI-X, TCQ, H/W RAID 0/1/10, ..
Hi Mark,
TCQ ? What about NCQ? :-)
Age
Quoted 1 line · age huisman
TCQ ? What about NCQ? :-)
Both kinds of SATA command-queuing will be included.\, whatever the name/abbreviation.
hdparm support as well (a first for SATA/SCSI).
Cheers
Signature · 2 lines
Mark Lord (hdparm keeper & the original "Linux IDE Guy")
Quoted 4 lines · J. Ryan Earl
On the brightside, md raid5 is often faster than hardware raid5. At least on the 7000 and 8000 series of 3ware hardware; the 9000 series looks promising though. I haven't seen megaraid SATA numbers, and I don't know what happened to the SX8.
Be cautious what you measure. One of he problems until you reach PCI-X is PCI bandwidth. Thus software md5 can look good but the moment its combined with other PCI activity goes down the pan entirely.
Quoted 2 lines
When the libata Marvell drivers come out, you'll have a cheap upgrade path for PCI-X boards if you want fast md raid:
Agreed. PCI-X will change a lot of this for boxes that are not very cpu/memory limited.
Quoted 5 lines · Alan Cox
Be cautious what you measure. One of he problems until you reach PCI-X is PCI bandwidth. Thus software md5 can look good but the moment its combined with other PCI activity goes down the pan entirely.
Right, which is why you'd think hardware based RAID would fair better, the parity information or mirror'd writes would only require one transfer/transaction across the PCI bus. However, when benchmarking a 3Ware 4-port ide raid controller (7000) we saw 40-50MB/sec read/write on 4x160GB drives raid5. With md raid5 (same controller, HDs, FS, etc) we saw 100MB/sec read, 60MB/sec write. This was using max 15% CPU on a 2.4GHz Pentium4, with a 32bit/33MHz PCI bus.
Quoted 10 lines
Quoted 5 lines
When the libata Marvell drivers come out, you'll have a cheap upgrade path for PCI-X boards if you want fast md raid:
Agreed. PCI-X will change a lot of this for boxes that are not very cpu/memory limited.
From the testing I've done, interconnect bandwidth has always been the limiting factor for the md driver. Using cheap ($~230) PCI-X motherboards--http://www.supermicro.com/products/motherboard/P4/E7210/P4SCT+.cfm--with dedicated gigE channels, you can make high density/price and performance/price NAS type appliances with no bottlenecks. For <$3K you can build a 2TB NAS server that'll keep a 250MB/s gigE link saturated.
-ryan
Quoted 8 lines · Jeff Garzik
As author of the current Linux SATA driver, I get the brunt of the questions and "bug reports" about "Linux doesn't support my hardware SATA RAID". Sigh. Silly marketing departments.
Thus, I have created a FAQ. You might sense a theme...
http://www.kernel.org/pub/linux/kernel/people/jgarzik/faq-sata-raid.html
Would be nice if somebody though could add how to compile dmraid with make-3.80 :/
Signature · 1 line
Martin Schlemmer