One Large md or Many Smaller md for Better Peformance?
Question: with the same number of physical drives, do I get better performance with one large md-based drive, or do I get better performance if I have several smaller md-based drives?
Situation: dual CPU, 4 drives (which I will set up as RAID-1 after being terrorized by the anti-RAID-5 polemics included in the Debian distro of mdadm).
I've two choices:
1. Allocate all the drive space into a single large partition, place into a single RAID array (either 10 or 1 + LVM, a separate question).
2. Allocate each drive into several smaller partitions. Make each set of smaller partitions into a separate RAID 1 array and use separate RAID md drives for the various file systems.
Example use case:
While working other problems, I download a large torrent in the background. The torrent writes to its own, separate file system called /foo. If /foo is mounted on its own RAID 10 or 1-LVM array, will that help or hinder overall system responsiveness?
It would seem a "no brainer" that giving each major filesystem its own array would allow for better threading and responsiveness, but I'm picking up hints in various piece of documentation that the performance can be counter-intuitive. I've even considered the possibility of giving /var and /usr separate RAID arrays (data vs. executables).
If an expert could chime in, I'd appreciate it a great deal.
Signature · 4 lines
Moshe Yudkowsky * [email protected] * www.pobox.com/~moshe "There are more ways to skin a cat than nuking it from orbit -- but it's the only way to be sure." -- Eliezer Yudkowsky
Quoted 3 lines · Moshe Yudkowsky
Question: with the same number of physical drives, do I get better performance with one large md-based drive, or do I get better performance if I have several smaller md-based drives?
No expert here, but my opinion:
- md code works better if it's only one array per physical drive, because it keeps statistics per array (like last accessed sector, etc.) and if you combine two arrays on the same drive these statistics are not exactly true anymore
- simply separating 'application work areas' into different filesystems is IMHO enogh, no need to separate the raid arrays too
- if you download torrents, fragmentation is a real problem, so use a filesystem that knows how to preallocate space (XFS and maybe ext4; for XFS use xfs_io to set a bigger extend size for where you download)
regards,
iustin
Thanks for the tips, and in particular:
Quoted 4 lines · Iustin Pop
- if you download torrents, fragmentation is a real problem, so use a filesystem that knows how to preallocate space (XFS and maybe ext4; for XFS use
xfs_ioto set a bigger extend size for where you download)
That's a very interesting idea; it also gives me an opportunity to experiment with XFS. I had been avoiding it because of possible power-failure issues on writes.
Signature · 4 lines
Moshe Yudkowsky * [email protected] * www.pobox.com/~moshe "She will have fun who knows when to work and when not to work." -- Segami
Moshe Yudkowsky ([email protected]) wrote on 20 January 2008 21:19:
Quoted 4 lines
Thanks for the tips, and in particular:
Iustin Pop wrote:
Quoted 8 lines
Quoted 4 lines
- if you download torrents, fragmentation is a real problem, so use a filesystem that knows how to preallocate space (XFS and maybe ext4; for XFS use
xfs_ioto set a bigger extend size for where you download)That's a very interesting idea; it also gives me an opportunity to experiment with XFS. I had been avoiding it because of possible power-failure issues on writes.
I use reiser3 and xfs. reiser3 is very good with many small files. A simple test shows interactively perceptible results: removing large files is faster with xfs, removing large directories (ex. the kernel tree) is faster with reiser3.
Quoted 4 lines · Carlos Carvalho
I use reiser3 and xfs. reiser3 is very good with many small files. A simple test shows interactively perceptible results: removing large files is faster with xfs, removing large directories (ex. the kernel tree) is faster with reiser3.
My current main concern about XFS and reiser3 is writebacks. The default mode for ext3 is "journal," which in case of power failure is more robust than the writeback modes of XFS, reiser3, or JFS -- or so I'm given to understand.
On the other hand, I have a UPS and it should shut down gracefully regardless if there's a power failure. I wonder if I'm being too cautious?
Signature · 4 lines
Moshe Yudkowsky * [email protected] * www.pobox.com/~moshe "Keep some secrets/Never tell, And they will keep you very well." -- Michelle Shocked
Quoted 8 lines · Moshe Yudkowsky
Quoted 4 lines · Carlos Carvalho
I use reiser3 and xfs. reiser3 is very good with many small files. A simple test shows interactively perceptible results: removing large files is faster with xfs, removing large directories (ex. the kernel tree) is faster with reiser3.
My current main concern about XFS and reiser3 is writebacks. The default mode for ext3 is "journal," which in case of power failure is more robust than the writeback modes of XFS, reiser3, or JFS -- or so I'm given to understand.
Also, barriers ("barrier=1" option for ext3) are not supported on filesystems placed on md/dm, it's a bit of a pain.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 11 lines · Moshe Yudkowsky
Quoted 4 lines · Carlos Carvalho
I use reiser3 and xfs. reiser3 is very good with many small files. A simple test shows interactively perceptible results: removing large files is faster with xfs, removing large directories (ex. the kernel tree) is faster with reiser3.
My current main concern about XFS and reiser3 is writebacks. The default mode for ext3 is "journal," which in case of power failure is more robust than the writeback modes of XFS, reiser3, or JFS -- or so I'm given to understand.
On the other hand, I have a UPS and it should shut down gracefully regardless if there's a power failure. I wonder if I'm being too cautious?
No.
If you haven't actually *tested* the UPS failover code to be sure your system is talking to the UPS properly, and that the UPS is able to hold up power long enough for a shutdown after the system detects the problem, then you don't know if you actually have protection. Even then, if you don't proactively replace batteries on schedule, etc, then you aren't as protected as you might wish to be.
And CPU fans fail, capacitors pop, power supplies fail, etc. These are things which have happened here in the last ten years. I also had a charging circuit in a UPS half-fail (from full wave rectifier to half). So the UPS would discharge until it ran out of power, then the system would fail hard. By the time I got on site and rebooted the UPS had trickle charged and would run the system. After replacing two "intermittent power supplies" in the system, the UPS was swapped on general principles and the real problem was isolated.
Shit happens, don't rely on graceful shutdowns (or recovery, have backups).
Signature · 3 lines
Bill Davidsen <[email protected]> "Woe unto the statesman who makes war without a reason that will still be valid when the war is over..." Otto von Bismark
Quoted 7 lines · Moshe Yudkowsky
Quoted 4 lines · Carlos Carvalho
I use reiser3 and xfs. reiser3 is very good with many small files. A simple test shows interactively perceptible results: removing large files is faster with xfs, removing large directories (ex. the kernel tree) is faster with reiser3.
Quoted 8 lines
My current main concern about XFS and reiser3 is writebacks. The default mode for ext3 is "journal," which in case of power failure is more robust than the writeback modes of XFS, reiser3, or JFS -- or so I'm given to understand.
On the other hand, I have a UPS and it should shut down gracefully regardless if there's a power failure. I wonder if I'm being too cautious?
I'm not sure what your actual worry is. It's not like XFS loses *commited* data on power failure. It may lose data that was never required to go to disk via fsync()/fdatasync()/sync. If someone is losing data on power failure is the unprotected write cache of the harddrive.
If you have properly-behaved applications, then they know when to do an fsync and if XFS returns success on fsync and your linux is properly configured (no write-back caches on drives that are not backed by NVRAM, etc.) then you won't lose data.
regards,
iustin
Quoted 8 lines · Moshe Yudkowsky
Question: with the same number of physical drives, do I get better performance with one large md-based drive, or do I get better performance if I have several smaller md-based drives?
Situation: dual CPU, 4 drives (which I will set up as RAID-1 after being terrorized by the anti-RAID-5 polemics included in the Debian distro of mdadm).
I've two choices:
1. Allocate all the drive space into a single large partition, place into a single RAID array (either 10 or 1 + LVM, a separate question).
One partitionable RAID-10, perhaps, then partition as needed. Read the discussion here about performance of LVM and RAID. I personally don't do LVM unless I know I will have to have great flexibility of configuration and can give up performance to get it. Other report different results, so make up your own mind.
Quoted 11 lines
2. Allocate each drive into several smaller partitions. Make each set of smaller partitions into a separate RAID 1 array and use separate RAID md drives for the various file systems.
Example use case:
While working other problems, I download a large torrent in the background. The torrent writes to its own, separate file system called /foo. If /foo is mounted on its own RAID 10 or 1-LVM array, will that help or hinder overall system responsiveness?
It would seem a "no brainer" that giving each major filesystem its own array would allow for better threading and responsiveness, but I'm picking up hints in various piece of documentation that the performance can be counter-intuitive. I've even considered the possibility of giving /var and /usr separate RAID arrays (data vs. executables).
If an expert could chime in, I'd appreciate it a great deal.
Signature · 3 lines
Bill Davidsen <[email protected]> "Woe unto the statesman who makes war without a reason that will still be valid when the war is over..." Otto von Bismark
Re: One Large md or Many Smaller md for Better Performance?
Quoted 1 line · Bill Davidsen
One partitionable RAID-10, perhaps, then partition as needed. Read the discussion here about performance of LVM and RAID. I personally don't do LVM unless I know I will have to have great flexibility of configuration and can give up performance to get it. Other report different results, so make up your own mind.
I've used Google to search (again) through the archives of the newsgroup, and "performance lvm" turns up relevant discussions back in 2004 or so, but nothing very recent. Am I missing some other location for discussions of this question, or perhaps I'm looking in the wrong places? (The Wiki didn't help either.)
Signature · 3 lines
Moshe Yudkowsky * [email protected] * www.pobox.com/~moshe "Rumor is information distilled so finely that it can filter through anything." -- Terry Pratchet, _Feet of Clay_
Quoted 1 line · Bill Davidsen
One partitionable RAID-10, perhaps, then partition as needed. Read the discussion here about performance of LVM and RAID. I personally don't do LVM unless I know I will have to have great flexibility of configuration and can give up performance to get it. Other report different results, so make up your own mind.
On MySQL servers I always always use LVM, even if performance is critical there: Snapshots! They make it easy and efficient to do an online snapshot after just freezing the database for a second or three.
- ask
[1] http://lenz.homelinux.org/mylvmbackup/
Signature · 1 line
http://develooper.com/ - http://askask.com/