ext2 vs ext3 on hppa
Hi all,
ext3 perf isn't so good on parisc.
I've posted profile data for kernel builds:
http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025196.html
But not for "tar". I can't find the data I collect before. :^( ISTR the culprit in ext3 was the bitmap search using atomic ops. On parisc, atomic ops are terrible since it requires a spinlock/op/unlock sequence.
I was asked about tar perf again and decided to post some results. If someone tells me which wiki page to add this, I'll pound parts of the data below into it.
In a nut shell, I timed unpacking the linux source tree:
FS real user sys
ext2 2m10.814s 0m53.424s 0m6.084s
ext3 3m46.577s 0m54.619s 1m34.553s
xfs 1m22.205s 0m53.952s 0m9.739s
md0+xfs 1m27.740s 0m54.550s 0m12.885s
User time is predominantly bzip2 running.
I didn't realize xfs can schedule work better than ext2 on a single disk. Adding md striping adds ~6% overhead...that's not so good. It's possible I need bigger chunks and that would go away.
I don't know if defaults are comparable between xfs and ext3. Is one implementing more or better features by default than the other?
I'll be looking at mdadm+xfs for future file systems to see if I can break it. Or I'll have to live without journaling on "stable" build machines. The ext3 "tar" performance is just unacceptable.
BTW, I get segfaults on the ext3 G
"raw" data follows.
grant
grundler@riot:~$ uname -a Linux riot 2.6.11-rc3-pa3 #6 SMP Mon Feb 7 04:25:07 UTC 2005 parisc64 GNU/Linux
grundler@riot:~$ cat /proc/cpuinfo processor : 0 cpu family : PA-RISC 2.0 cpu : PA8700 (PCX-W2) cpu MHz : 650.000000 model : 9000/800/A500-6X ... processor : 1 cpu family : PA-RISC 2.0 cpu : PA8700 (PCX-W2) cpu MHz : 650.000000 model : 9000/800/A500-6X model name : Crescendo 650 W2 ...
riot:/# lsscsi
[1:0:15:0] disk SEAGATE ST336704LC HP01 /dev/sda
[4:0:0:0] disk SEAGATE ST318451LC HP00 /dev/sdb
[4:0:1:0] disk SEAGATE ST318451LC HP01 /dev/sdc
[4:0:2:0] disk SEAGATE ST318451LC HP00 /dev/sdd
[4:0:3:0] disk SEAGATE ST318451LC HP01 /dev/sde
[4:0:8:0] disk SEAGATE ST318451LC HP00 /dev/sdf
[4:0:9:0] disk SEAGATE ST318451LC HP00 /dev/sdg
[4:0:10:0] disk SEAGATE ST318451LC HP00 /dev/sdh
[4:0:12:0] disk SEAGATE ST318451LC HP01 /dev/sdi
[4:0:13:0] disk SEAGATE ST318451LC HP01 /dev/sdj
[4:0:14:0] enclosu HP A5272A HP04 -
[5:0:0:0] disk SEAGATE ST318451LC HP01 /dev/sdk
[5:0:1:0] disk SEAGATE ST318451LC HP00 /dev/sdl
[5:0:2:0] disk SEAGATE ST318451LC HP00 /dev/sdm
[5:0:3:0] disk SEAGATE ST318451LC HP00 /dev/sdn
[5:0:8:0] disk SEAGATE ST318451LC HP00 /dev/sdo
[5:0:9:0] disk SEAGATE ST318451LC HP00 /dev/sdp
[5:0:10:0] disk SEAGATE ST318451LC HP00 /dev/sdq
[5:0:11:0] disk SEAGATE ST318451LC HP00 /dev/sdr
[5:0:12:0] disk SEAGATE ST318451LC HP01 /dev/sds
[5:0:13:0] disk SEAGATE ST318451LC HP01 /dev/sdt
[5:0:14:0] enclosu HP A5272A HP05 -
riot:/# lspci -vt
-+-[30]-+-02.0 LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter
| \-02.1 LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter
+-[20]---00.0 LSI Logic / Symbios Logic 53c1010 66MHz Ultra3 SCSI Adapter
+-[10]---00.0 Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet
\-[00]-+-00.0 Digital Equipment Corporation DECchip 21142/43
+-01.0 LSI Logic / Symbios Logic 53C896/897
+-01.1 LSI Logic / Symbios Logic 53C896/897
+-02.0 LSI Logic / Symbios Logic 53c875
+-02.1 LSI Logic / Symbios Logic 53c875
+-04.0 Hewlett-Packard Company: Unknown device 128d
\-04.1 Hewlett-Packard Company Diva Serial [GSP] Multiport UART
...# mkfs /dev/sdk mke2fs 1.35 (28-Feb-2004) /dev/sdk is entire device, not just one partition! Proceed anyway? (y,n) y Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 2223872 inodes, 4445810 blocks 222290 blocks (5.00%) reserved for the super user First data block=0 136 block groups 32768 blocks per group, 32768 fragments per group 16352 inodes per group ... riot:/mnt# time tar xjf linux-2.6.11-rc3-pa3.tar.bz2
real 2m10.814s user 0m53.424s sys 0m6.084s
riot:/mnt#
...
riot:/# mke2fs -j /dev/sdk mke2fs 1.35 (28-Feb-2004) /dev/sdk is entire device, not just one partition! Proceed anyway? (y,n) y Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 2223872 inodes, 4445810 blocks 222290 blocks (5.00%) reserved for the super user First data block=0 136 block groups 32768 blocks per group, 32768 fragments per group 16352 inodes per group ... riot:/# mount /dev/sdk /mnt riot:/# cd /mnt riot:/mnt# time tar xjf ~/linux-2.6.11-rc3-pa3.tar.bz2
real 3m46.577s user 0m54.619s sys 1m34.553s
riot:/mnt#
riot:/# mkfs -t xfs /dev/sdk
meta-data=/dev/sdk isize=256 agcount=16, agsize=277863 blks
= sectsz=512
data = bsize=4096 blocks=4445808, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=2560, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0riot:/mnt# time tar xjf ~/linux-2.6.11-rc3-pa3.tar.bz2
real 1m22.205s user 0m53.952s sys 0m9.739s
riot:/# mdadm -C /dev/md0 -l 0 -n 6 /dev/sdb /dev/sdk /dev/sdc /dev/sdl /dev/sdd /dev/sdm
...
riot:/# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1]
md0 : active raid0 sdm[5] sdd[4] sdl[3] sdc[2] sdk[1] sdb[0]
106699008 blocks 64k chunks
...
riot:/# mkfs -t xfs /dev/md0
meta-data=/dev/md0 isize=256 agcount=16, agsize=1667184 blks
= sectsz=512
data = bsize=4096 blocks=26674752, imaxpct=25
= sunit=16 swidth=96 blks, unwritten=1
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=13024, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=393216 blocks=0, rtextents=0
riot:/# mount /dev/md0 /mnt
riot:/# cd /mnt
riot:/mnt# time tar xjf ~/linux-2.6.11-rc3-pa3.tar.bz2 real 1m27.740s user 0m54.550s sys 0m12.885s
[ after running some kernel builds on the xfs file system ]
grundler@riot:/mnt$ time tar xjf /root/linux-2.6.11-rc3-pa3.tar.bz2
real 2m35.000s user 0m55.749s sys 0m21.332s
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 2 lines · Grant Grundler
Hi all,
ext3 perf isn't so good on parisc.
Well, From what I know on the linux filesystem types, I can imagine that ext3 is not the fastest among them.
ext2 is a (the?) linux normal filesystem: data storage is straigt-forward and such.
ext3 is basically ext2 with added journalling. For reading, it should be just as fast as ext2. For writing it is definitly slower than ext2 since it does ext2 and journalling. As far as I know, the journalling is added to get additional data-safety; the filesystem-check is definitly faster/better than for ext2. I expect this is at the cost of writing performance.
I don't know xfs, I expect it to be comparable with reiserfs in that it is a journalling filesystem, designed from scratch. For these filesystems, I expect better performance than ext3 and, depending on several parameters, even better performance than ext2.
Between ext2 and xfs/reiser I expect ext2 to be better for large files and the others better for lots of small files.
Quoted 13 lines
I've posted profile data for kernel builds:
http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025196.html
But not for "tar". I can't find the data I collect before. :^( ISTR the culprit in ext3 was the bitmap search using atomic ops. On parisc, atomic ops are terrible since it requires a spinlock/op/unlock sequence.
I was asked about tar perf again and decided to post some results. If someone tells me which wiki page to add this, I'll pound parts of the data below into it.
Quoted 7 lines
In a nut shell, I timed unpacking the linux source tree:
FS real user sys
ext2 2m10.814s 0m53.424s 0m6.084s
ext3 3m46.577s 0m54.619s 1m34.553s
xfs 1m22.205s 0m53.952s 0m9.739s
md0+xfs 1m27.740s 0m54.550s 0m12.885s
This clearly shows, that the filesystems do have some differences in performance and all are in system-time, the user-time is comparable.
It would be nice if you can give similar stats for different architectures, either within pa-risc or even on other platforms.
Quoted 2 lines
User time is predominantly bzip2 running.
There are several things in comparing filesystems. Some (ext2 I expect) filesystems are good in hanldling large files. Some (I expect the journaling ones) are good in handling many small files.
With bzip2 running, I expect you use packing or unpacking the kernel-source for a test. To get raw figures, better use tar, cp -R or dd. And be sure to work from one disk to an other.
Quoted 4 lines
I didn't realize xfs can schedule work better than ext2 on a single disk. Adding md striping adds ~6% overhead...that's not so good. It's possible I need bigger chunks and that would go away.
I don't know what you mean by mdstripping, If it is something like raid-0 or raid-1 (or the combination) then striping is only faster on reading, not on writing.
Quoted 3 lines
I don't know if defaults are comparable between xfs and ext3. Is one implementing more or better features by default than the other?
As above: ext3 is ext2 with added journalling. xfs is designed from the ground up. I'm surprised to see that ext3 takes 15 times more system-time!
Quoted 5 lines
I'll be looking at mdadm+xfs for future file systems to see if I can break it. Or I'll have to live without journaling on "stable" build machines. The ext3 "tar" performance is just unacceptable.
You'd definitly also peek at reiserfs, Its the default fs used in SuSE linux.
If you intend to re-do your tests, think about the next: To test read speed, cp -r to /dev/null. To test write speed of small files, tar from the fastest read-system. (for large files, just copy from /dev/zero).
Quoted 2 lines
BTW, I get segfaults on the ext3 G
Thats a bad sign...
my 2 cents
CBee
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
On Fri, Feb 11, 2005 at 02:08:12PM +0100, Corn? Beerse wrote:
...
Quoted 7 lines
Quoted 7 lines
In a nut shell, I timed unpacking the linux source tree:
FS real user sys
ext2 2m10.814s 0m53.424s 0m6.084s
ext3 3m46.577s 0m54.619s 1m34.553s
xfs 1m22.205s 0m53.952s 0m9.739s
md0+xfs 1m27.740s 0m54.550s 0m12.885s
...
Quoted 2 lines
It would be nice if you can give similar stats for different architectures, either within pa-risc or even on other platforms.
Sorry, who ever is interested in that comparison can provide that data. :^)
Quoted 3 lines
I don't know what you mean by mdstripping, If it is something like raid-0 or raid-1 (or the combination) then striping is only faster on reading, not on writing.
Yes, I meant raid0.
Quoted 2 lines
You'd definitly also peek at reiserfs, Its the default fs used in SuSE linux.
That sounds like an excercise for someone else.
Quoted 3 lines
If you intend to re-do your tests, think about the next: To test read speed,
cp -rto/dev/null. To test write speed of small files, tar from the fastest read-system. (for large files, just copy from/dev/zero).
ok. My guess was with 4GB of RAM, I was mostly operating from buffer cache. Maybe next time I'll use ram disks instead.
Quoted 3 lines
Quoted 1 line
BTW, I get segfaults on the ext3.
Thats a bad sign...
My bad. I was running two instances of "make" in the same source tree. I had forgotten about the "while :" loop that was doing builds. /me hides
thanks,
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
[...]
Quoted 5 lines
My bad. I was running two instances of "make" in the same source tree. I had forgotten about the "while :" loop that was doing builds. /me hides
mmm when I did (sorry not any more able to do it :_( ) similar test it seems that segfault to only occured with option -j2 (or greater) and only on smp system (on such b2k the same test _never_ segfault also on ext3)
hth,
Joel
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 10 lines · Grant Grundler
Hi all,
ext3 perf isn't so good on parisc.
I've posted profile data for kernel builds:http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025196.html
But not for "tar". I can't find the data I collect before. :^( ISTR the culprit in ext3 was the bitmap search using atomic ops. On parisc, atomic ops are terrible since it requires a spinlock/op/unlock sequence.
I switched the ext2/3 bitops to use the non-atomic ops. Tried it on nicol (dual 550MHz PA8600) ...
$ time tar xjf ~/linux-2.6.11-rc3.tar.bz2
real 25m4.557s user 1m7.262s sys 24m59.092s
ouch. time to do some profiling, i guess, instead of assuming I saw the problem and the fix ...
Signature · 10 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain _______________________________________________ parisc-linux mailing list [email protected] http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 1 line · Grant Grundler
But not for "tar". I can't find the data I collect before. :^(
I found them:
-rw-r--r-- 1 root root 33517 Jan 9 21:27 prof-2.6.10-pa6-01 -rw-r--r-- 1 root root 23770 Jan 16 19:46 prof-2.6.10-pa6-02
and posted both on
http://www.parisc-linux.org/~grundler/prof-a500-65/
Quoted 3 lines
ISTR the culprit in ext3 was the bitmap search using atomic ops. On parisc, atomic ops are terrible since it requires a spinlock/op/unlock sequence.
Yeah, this seems to be right.
I gather this is only an issue for SMP kernels.
grundler <506>sort -rnk 3 prof-2.6.10-pa6-01
55063 _spin_lock 1376.5750
42958 ext3_test_allocatable 97.6318 5974 rb_next 57.4423 5541 flush_kernel_icache_page 55.4100 800 fdsync 40.0000 1279 flush_user_icache_range_asm 35.5278 35341 bitmap_search_next_usable_block 30.6780 1328 _spin_lock_irqsave 27.6667 1470 machine_restart 22.9688 747 flush_user_dcache_range_asm 20.7500 1043 cpu_idle 14.4861
135 fisync 8.4375
647 __clear_user_page_asm 5.7768
...
Sorry - I didn't note exactly which workload the above represented.
And I suspect -02 data is with willy's vma_prev patch applied that he proposed here:
http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025210.html
grundler <509>sort -rnk 3 prof-2.6.10-pa6-02
61105 _spin_lock 1527.6250
43040 ext3_test_allocatable 97.8182 4902 rb_next 47.1346 35646 bitmap_search_next_usable_block 30.9427 2217 flush_kernel_icache_page 22.1700 969 _spin_lock_irqsave 20.1875 1055 machine_restart 16.4844 764 cpu_idle 10.6111 157 fdsync 7.8500 270 flush_user_icache_range_asm 7.5000 209 flush_user_dcache_range_asm 5.8056
...
hth,
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 3 lines · Grant Grundler
And I suspect -02 data is with willy's
vma_prevpatch applied that he proposed here:
http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025210.html
Probably not -- it crashes the machine ;-)
Signature · 10 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain _______________________________________________ parisc-linux mailing list [email protected] http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
Quoted 6 lines · Matthew Wilcox
Quoted 3 lines · Grant Grundler
And I suspect -02 data is with willy's
vma_prevpatch applied that he proposed here:
http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025210.htmlProbably not -- it crashes the machine ;-)
Ok :^(
We called flush_kernel_icache_page half as many times (5541 vs 2217). And flush_user_icache_range_asm 1/5th a often (1279 vs 270). Both are SMP kernels.
I have no idea what else might result in such a major difference.
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux