xfs for free hosting on linux ? performance questions
hello, i'm going to create xfs on RAID 5 array consiting of 5 drives, each 500GB, it will be software raid made with mdadm on linux. i run free hosting, and the array is only for users files (system and such are on ext3).
what options do you recommend while creating and while mounting? i had performance problems while testing this setup with bonnie++ and postmark (postmark especially).
data will be in many directories, users files are in /var/www/users/username and i have more than 100 000 accounts. also a policy to make max 3 MB files, so most of my files are around 50 KB i think.
so knowing this all, what would you advice me if i may ask? my current idea was like this:
mkfs.xfs -l size=128m,lazy-count=1
mount -o nobarrier,noatime,logbufs=8,logbsize=256k
this will have full backup, so i'm more after performance, not stability, so if you can give me some performance tips... (postmark run on array made from 4 disks, not 5, durning test, with parameters - number of files 10k, transactions 10k, subdirs 20k, was giving very very slow reads, around 200KB/s, while reiser4/btrfs were like 2MB/s reads, but even ext3 was faster - any idea why? i tested on 2.6.24.3 and 2.6.25-rc4 )
i keep seeing tests like this: http://www.t2-project.org/zine/1/
also, i have real life example, that 3 disks RAID-0 array with tweaked xfs is as fast as just one disk with reiser4 (my free hosting with millions of files)
i know reiser4 is not stable (i also experienced this), so i'm asking if
you know any additional tips to tweak xfs, or maybe you know why it
has performance problems in some situations.
thanks in advance
[ ...
Quoted 4 lines
data will be in many directories, users files are in
/var/www/users/usernameand i have more than 100 000 accounts. also a policy to make max 3 MB files, so most of my files are around 50 KB i think.
This is a much better situation than the frequent inane one where someone asks about using a filesystem as a database for small records.
But a ''source files'' filesystem is still a bit of challenge.
Quoted 2 lines
[ ... ] mount -o nobarrier,noatime,logbufs=8,logbsize=256k [ ... ]
Even with backups, 'nobarrier' means begging for trouble.
Quoted 7 lines
[ ... ] number of files 10k, transactions 10k, subdirs 20k, was giving very very slow reads, around 200KB/s, while reiser4/btrfs were like 2MB/s reads, but even ext3 was faster
- any idea why? i tested on 2.6.24.3 and 2.6.25-rc4 ) [ ... ] that 3 disks RAID-0 array with tweaked xfs is as fast as just one disk with reiser4 (my free hosting with millions of files) [ ... ]
Just use the by now very well tested ReiserFS version 3 for that. Just as you would use XFS for bulk streaming of large files.
Recently on the same 4x(1+1) RAID10 f2 I did some (single threaded, single file reading and writing) tests involving various file systems with both large files (2-12 2GB ones) and a directory tree contain lots of small Java source files (6.5GB in 50k directories containing 150k files).
While most file systems could reach something like 250MB/s writing and 430MB/s reading with the small set of large files (with XFS a few dozen MB/s ahead of most of the others and would have been more so on multithreaded), on the directory tree ReiserFS did around 57MB/s, JFS around 36MB/s, and XFS around 20-25MB/s.
There are several reasons why ReiserFS is better suited for large directory tree with lots of small files; one of them is that it was designed for that :-).
Quoted 4 lines
There are several reasons why ReiserFS is better suited for large directory tree with lots of small files; one of them is that it was designed for that :-).
but reiserfs is slow for me too, maybe i have some other problems? i made array like this mdadm --create /dev/md0 --verbose --level=5 --raid-devices=5 /dev/sd{b,c,d,e,f}1 --assume-clean
then just mkfs.reiserfs /dev/md0 and then mount /dev/md0 to /array and run postmark with these:
set numbers 20000
set transactions 10000
set subdirectories 20000
set location /array/
results i got:
Time:
287 seconds total
118 seconds of transactions (84 per second)
Files:
25014 created (87 per second)
Creation alone: 20000 files (168 per second) Mixed with transactions: 5014 files (42 per second)
5026 read (42 per second)
4971 appended (42 per second)
25014 deleted (87 per second)
Deletion alone: 20028 files (400 per second) Mixed with transactions: 4986 files (42 per second)
Data:
26.48 megabytes read (94.47 kilobytes per second) 136.18 megabytes written (485.89 kilobytes per second)
is there something wrong with my system?? LA is 0.50 right now (i'm testing it at night), hdparm is showing that every HD from the array is doing 100MB/s at least, so why these numbers?
[[HTML alternate version deleted]]
Quoted 5 lines
but reiserfs is slow for me too, maybe i have some other problems? i made array like this mdadm --create
/dev/md0--verbose --level=5 --raid-devices=5/dev/sd{b,c,d,e,f}1--assume-clean
Uhhhhh, '--assume-clean' on a RAID5 is not necessarily a good idea. Just don't do it, and wait a few hours for the initial sync to happen.
Quoted 5 lines
[ ... ]
26.48 megabytes read (94.47 kilobytes per second) 136.18 megabytes written (485.89 kilobytes per second)is there something wrong with my system??
The speed above is indeed terrible. Perhaps the array is rebuilding frantically as it finds all stripes have the wrong parity because of '--assume-clean'.
Quoted 1 line
[ ... ] LA is 0.50 right now (i'm testing it at night),
Why is LA relevant? What is running on that system now? Are those physical disks being used by some applications?
Quoted 2 lines
hdparm is showing that every HD from the array is doing 100MB/s at least, so why these numbers?
Perhaps after rebuilding the array without '--assume-clean' you might try to do such simple tests on the '/dev/md0' device itself, just to be sure that it performs more or less adequately.
After several reports that is helps, try this before the test:
blockdev --setra 1024 /dev/md0
but even without it you should be getting at least 40-50MB/s.
And always check the array status with 'mdadm --detail' to see if it is resyncing or not before testing.
Using 'watch iostat -k sd{b,c,d,e,f} 1 2' (and checking out the _second_ set of figures) is also nice to see the actual transfer rates of each drive in the array and verify that if you are reading there is no writing going on etc.