LVM on SATA/PATA disks
Dear all,
I always have been told (S/P)ATA disks should somehow be avoided in server systems. This mainly because the high rate of interrupt requests generated by the IO subsystem will constantly interrupt the CPU which then becomes unavailable for other server tasks. SCSI disks should be preferred mainly for that reason.
Can someone confirm this statement?
FYI, we have to upgrade one of our development machine - hence the question. This machine is mainly used to run unattended software builds -> moves large quantity of small files and requires lots of CPU.
(sorry if my question is "stupid" or has already been asked many times - but couldn't find any hint anywere)
Thanks
Quoted 7 lines · Bertrand Renuart
I always have been told (S/P)ATA disks should somehow be avoided in server systems. This mainly because the high rate of interrupt requests generated by the IO subsystem will constantly interrupt the CPU which then becomes unavailable for other server tasks. SCSI disks should be preferred mainly for that reason.
Can someone confirm this statement?
The interrupt rate has nothing to do with the type of disk, and a lot to do with the controller. There is a CPU difference between $50 consumer IDE/SATA adapters, and $300 server grade IDE/SATA adapters. You'll want the controller to support fast DMA at minimum.
Quoted 3 lines
FYI, we have to upgrade one of our development machine - hence the question. This machine is mainly used to run unattended software builds -> moves large quantity of small files and requires lots of CPU.
However, even the $50 adapters are way faster than what you could get 10 years ago. I would start with a higher end $50+ adapter (you want it reliable). If you get a lot of "system" CPU time during heavy disk IO, then the $300 adapter will relieve that. For example, a super cheap ATA adapter was giving us 25% system CPU on backups on a 1.8Ghz Celeron. Replacing that with a more expensive (but < $100) IDE adapter reduced system time to 1% or so.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 5 lines · Stuart D. Gathman
The interrupt rate has nothing to do with the type of disk, and a lot to do with the controller. There is a CPU difference between $50 consumer IDE/SATA adapters, and $300 server grade IDE/SATA adapters. You'll want the controller to support fast DMA at minimum.
I thought the biggest thing that SCSI had that IDE didn't was SCSI's ability to shovel an ass-barn-load of data to a disk and the disk would go deal with it, giving up the SCSI bus so that another disk could be shovelled another ass-barn-load of data to go and deal with, and so on.
Of course one has to remember it takes much much more time for the disk to actually write (or read as the case may be) the data to media than it takes to shovel the data to the disk (which it would cache locally before writing to actual media) on the bus.
The contrast with IDE (or PATA as I guess the trendy name is), again as I always thought was that the IDE bus was not available for use while a disk was still pending a media I/O operation, so that with multiple devices, you could not leverage the I/O of the IDE bus using multiple devices, essentially in parallel. I guess this is where having system with multiple IDE buses and only putting a single device per bus grew from.
Of course on a SCSI bus you can put many many disks -- to the capacity of the bus -- or more, but that would be silly.
How does SATA fit in with all of this? Is it basically the same limitations on the bus as IDE/PATA, so that you'd really not want to put more than 1 device per bus?
b.
Signature · 3 lines
My other computer is your Microsoft Windows server. Brian J. Murrell
Quoted 14 lines · Brian J. Murrell
Quoted 5 lines · Stuart D. Gathman
The interrupt rate has nothing to do with the type of disk, and a lot to do with the controller. There is a CPU difference between $50 consumer IDE/SATA adapters, and $300 server grade IDE/SATA adapters. You'll want the controller to support fast DMA at minimum.
I thought the biggest thing that SCSI had that IDE didn't was SCSI's ability to shovel an ass-barn-load of data to a disk and the disk would go deal with it, giving up the SCSI bus so that another disk could be shovelled another ass-barn-load of data to go and deal with, and so on.
...
Quoted 7 lines
The contrast with IDE (or PATA as I guess the trendy name is), again as I always thought was that the IDE bus was not available for use while a disk was still pending a media I/O operation, so that with multiple devices, you could not leverage the I/O of the IDE bus using multiple devices, essentially in parallel. I guess this is where having system with multiple IDE buses and only putting a single device per bus grew from.
We always put exactly one IDE disk per channel for that very reason. You are correct that with 2 disks on the same channel, only one can be active at a time. So don't do that. A $50 IDE PCI card give you 2 IDE channels - for 2 disks in high performance mode (suitable for mirroring). Buy two cards for 4 disks. Or a $300 server card for 8 disks.
Quoted 3 lines
How does SATA fit in with all of this? Is it basically the same limitations on the bus as IDE/PATA, so that you'd really not want to put more than 1 device per bus?
SATA mandates at most 1 disk per channel, making the issue moot. It is still true that there is only one active disk on a bus. But then there is only one disk on a bus.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 7 lines · Stuart D. Gathman
Quoted 3 lines
How does SATA fit in with all of this? Is it basically the same limitations on the bus as IDE/PATA, so that you'd really not want to put more than 1 device per bus?
SATA mandates at most 1 disk per channel, making the issue moot. It is still true that there is only one active disk on a bus. But then there is only one disk on a bus.
Newer SATA drives, with proper newer controllers and proper device driver support will support NCQ (native command queueing), which allows the drive to re-order requests. It appears that the Linux ACHI and Nvidia SATA drivers support this capability in recent kernels.
Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires filesystem and driver support of write barriers for reliability. Write barriers are not implement in DM, hence LVM, so there is a reliability risk in going with this kind of solution. Depending on the filesystem this can result in power failures resulting in files having inconsistent data.
Dave
Quoted 14 lines · David Brown
Quoted 6 lines · Stuart D. Gathman
Quoted 3 lines
How does SATA fit in with all of this? Is it basically the same limitations on the bus as IDE/PATA, so that you'd really not want to put more than 1 device per bus?
SATA mandates at most 1 disk per channel, making the issue moot. It is still true that there is only one active disk on a bus. But then there is only one disk on a bus.
Newer SATA drives, with proper newer controllers and proper device driver support will support NCQ (native command queueing), which allows the drive to re-order requests. It appears that the Linux ACHI and Nvidia SATA drivers support this capability in recent kernels.
Quoted 6 lines
Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires filesystem and driver support of write barriers for reliability. Write barriers are not implement in DM, hence LVM, so there is a reliability risk in going with this kind of solution. Depending on the filesystem this can result in power failures resulting in files having inconsistent data.
Are you saying that LVM on SCSI is not safe in this scenario?
Signature · 2 lines
Les Mikesell [email protected]
Quoted 8 lines · Les Mikesell
Quoted 6 lines
Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires filesystem and driver support of write barriers for reliability. Write barriers are not implement in DM, hence LVM, so there is a reliability risk in going with this kind of solution. Depending on the filesystem this can result in power failures resulting in files having inconsistent data.
Are you saying that LVM on SCSI is not safe in this scenario?
If out of order writes are enabled, then your server should hold power to the disk drives for part of a second after disabling further writes in software. LVM is a low risk because LVM changes are comparatively rare, and the modification window is small. But, theoretically, if you lost power right at the instant you pressed return on lvcreate, lvextend, or lvremove, on a system with busy disk io, you could corrupt the LVM metadata.
Having a UPS and doing a shutdown solves the problem. However, if your server enables out of order writes with native queuing, and loses power without a shutdown, and doesn't keep drives powered long enough after stopping the CPUs (this is why high end servers now have a microprocessor dedicated to power sequencing), then you should not rely on filesystem journalling, and should do an fsck at reboot. FS journals rely on physical writes taking place in-order, or at least having 'sync' calls to finish previous logical writes before starting a new one (write barrier). There is a similar issue with creating a database on top of a file. For example, you have to use 'fsync' after writing a checkpoint, before beginning updates.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 7 lines · Stuart D. Gathman
If out of order writes are enabled, then your server should hold power to the disk drives for part of a second after disabling further writes in software. LVM is a low risk because LVM changes are comparatively rare, and the modification window is small. But, theoretically, if you lost power right at the instant you pressed return on lvcreate, lvextend, or lvremove, on a system with busy disk io, you could corrupt the LVM metadata.
LVM metadata I/O does not pass through device-mapper unless you are stacking volume groups so I'm not sure how your comment regarding barriers is relevant here. It's also only going to affect devices under snapshot and multipath mappings.
How does this differ from updating an MSDOS or GPT partition table, for example?
Kind regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFGSB/v6YSQoMYUY94RAubIAJ4vcB/tqherGhmW/M6n1wcO5KGungCffw1w ISfLoRVk6CP3EryIzHTolXA=
=g18w
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 3 lines · David Brown
Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires filesystem and driver support of write barriers for reliability. Write barriers are not implement in DM, hence LVM, so there is a
Only the dm-mpath and dm-snap targets treat barrier bios specially (they return EOPNOTSUPP). Other dm targets pass the barriers on to the underlying devices unchanged.
Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFGSB4y6YSQoMYUY94RApajAJ4kyrw+DDEjj3WiXPOysO5J6KKmOQCgxC0M oQrw2KA74j+ytiKFoMLPQdg=
=CZvy
-----END PGP SIGNATURE-----
Quoted 8 lines · Bryn M. Reeves
Quoted 3 lines · David Brown
Of course, any of the re-ordering (SCSI TCQ, or SATA NCQ) requires filesystem and driver support of write barriers for reliability. Write barriers are not implement in DM, hence LVM, so there is a
Only the dm-mpath and dm-snap targets treat barrier bios specially (they return EOPNOTSUPP). Other dm targets pass the barriers on to the underlying devices unchanged.
A simple lvm volume, mounted with XFS gives the following kernel message:
Filesystem "dm-11": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem dm-11
so I'm fairly sure that the write barriers don't work. This is the 2.6.20 kernel. XFS is the only filesystem nice enough to warn you that the result might not be as reliable.
Dave
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 6 lines · David Brown
A simple lvm volume, mounted with XFS gives the following kernel message:
Filesystem "dm-11": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem dm-11
So it does. I don't use XFS on my boxes & hadn't noticed this before.
Turns out that xfs is checking barrier capabilities using queue->ordered in the device struct. If it has QUEUE_ORDERED_NONE, barriers are disabled (fs/xfs/linux-2.6/xfs_super.c).
I'll look into why this is the case as given this check there shouldn't be any need for snap & mpath to check for barrier bios themselves.
The patches that introduced those checks were submitted after problems were seen with barriers on these target types, so something was clearly submitting them at some stage.
Kind regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFGScUv6YSQoMYUY94RAh3sAKCpcTdyvLKpwni1VYssyo2cORBG6ACbBrbl xlJIlDXDdZGNAX5Tmjbt7RI=
=yaL7
-----END PGP SIGNATURE-----
Quoted 8 lines
However, even the $50 adapters are way faster than what you could get 10 years ago. I would start with a higher end $50+ adapter (you want it reliable). If you get a lot of "system" CPU time during heavy disk IO, then the $300 adapter will relieve that. For example, a super cheap ATA adapter was giving us 25% system CPU on backups on a 1.8Ghz Celeron. Replacing that with a more expensive (but < $100) IDE adapter reduced system time to 1% or so.
What adapter would you recommend for a descent setup?
Quoted 2 lines · Bertrand Renuart
What adapter would you recommend for a descent setup?
The <$100 types keep changing availability. Here are the lspci ids of some <$100 IDE adapters that are working great with low overhead (mirroring with 2 disks attached, one on each channel):
105a:4d30 Promise Technology, Inc. 20267 (rev 02)
Subsystem: Promise Technology, Inc. Ultra100 (ATA100 is old, I know)
1095:0680 CMD Technology Inc PCI0680 (rev 01)
1095:0680 Silicon Image, Inc. PCI0680 Ultra ATA-133 Host Controller (rev 02)
Subsystem: Silicon Image, Inc. PCI0680 Ultra ATA-133 Host Controller
(tricky to boot from due to wierd BIOS drive assignments)
I've bought 2 IDE PCI cards that didn't work well. I pitched them and did not record the numbers. Shame on me, or maybe QA isn't perfect on the cheap cards and I got a bum unit.
I haven't tried any add-on SATA boards yet.
Just buy one and try it. Taiwan is better than China.
If you need something repeatable, you'll have to go with a $300 server kind such as what Dell sells.
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
For linux, I prefer anything 3ware makes for SATA or PATA. I have not been happy with promise boards. The newer higher end boards are expensive, but you can buy the older boards for significantly less. With the 3ware boards you could even avoid having to use lvm since they have raid included.
Dan
Quoted 17 lines · Bertrand Renuart
Quoted 8 lines
However, even the $50 adapters are way faster than what you could get 10 years ago. I would start with a higher end $50+ adapter (you want it reliable). If you get a lot of "system" CPU time during heavy disk IO, then the $300 adapter will relieve that. For example, a super cheap ATA adapter was giving us 25% system CPU on backups on a 1.8Ghz Celeron. Replacing that with a more expensive (but < $100) IDE adapter reduced system time to 1% or so.
What adapter would you recommend for a descent setup?
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/