benchmark: write barrier/write cache on XFS

DISCUSSION Martin Steigerwald 1 participant 2 messages Dec 8, 2008 VFS

lore

Share & tools

lore

Martin Steigerwald

author

2008-12-08 20:40 UTC #0 lore raw

Hi!

I got curious about the recent discussions about the write barrier feature[1][2].

Thus I did my own benchmark this evening. Since I use XFS and tested with XFS for now only. Write barrier + write cache, no barrier + write cache, no write barrier + no write cache. I just did tar -xf linux-2.6.27.tar.bz2 and rm -rf linux-2.6.27.

My conclusion is: At least for this metadata intensive workload enabling write barries is complete nonsense for XFS filesystems, cause it runs way faster without write cache and without barriers. I am completely puzzled about this, cause I always thought that barriers where meant to provide a performance improvement versus disabling write cache. I actually advertised them as such in my Linux magazine article, should have benched them before it seems. Write barrier seem to be a slow down feature for XFS.

See for yourself. I am interested in other benchmarks like this. Automating this would be good and running it for different filesystems. Maybe adding some different workloads as these are highly selective tests, testing with RAID systems as Justin did and of course testing with different filesystems.

I think I will disable write cache and disable barriers on that ThinkPad T42 and I think also other machines.

IBM ThinkPad T42 with 160 GB IDE hitachi drive via libata:

shambhala:~> grep "model name" /proc/cpuinfo
model name : Intel(R) Pentium(R) M processor 1.80GHz
shambhala:~> cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ondemand
shambhala:~> cat /proc/version
Linux version 2.6.27.7-tp42-toi-3.0-2008-11-25 (martin@shambhala) (gcc version 4.3.2 (Debian 4.3.2-1) ) #1 PREEMPT Sun Nov 30 10:29:09 CET 2008

shambhala:~> hdparm -I /dev/sda | egrep "(Model Num|device size|power manage|acoustic manage|DMA:|120 ns)"
Model Number: Hitachi HTS541616J9AT00
device size with M = 1024*1024: 152627 MBytes
device size with M = 1000*1000: 160041 MBytes (160 GB)
Advanced power management level: 254
Recommended acoustic management value: 128, current value: 128
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5

The default - write barrier and write cache enabled:

shambhala:~> hdparm -W /dev/sda

/dev/sda:
write-caching = 1 (on)
shambhala:~> hdparm -I /dev/sda | grep cache
* Write cache
shambhala:~> mkfs.xfs -f -L xfs -l lazy-count=1 /dev/sda6
meta-data=/dev/sda6 isize=256 agcount=4, agsize=1344188
blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=5376750, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=2625, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

shambhala:~> mount /dev/sda6 /mnt/zeit
shambhala:~> grep zeit /proc/mounts
/dev/sda6 /mnt/zeit xfs rw,attr2,noquota 0 0
shambhala:~> cd /mnt/zeit
shambhala:/mnt/zeit> sync ; time
tar -xf /home/martin/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2; time sync
tar -xf /home/martin/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 46,01s user 5,59s system 41% cpu 2:03,01 total
sync 0,00s user 0,18s system 3% cpu 5,303 total shambhala:/mnt/zeit> sync ; time rm -rf linux-2.6.27 ; time sync rm -rf linux-2.6.27 0,08s user 3,49s system 6% cpu 51,482 total sync 0,00s user 0,15s system 45% cpu 0,320 total shambhala:/mnt/zeit> cd
shambhala:~> umount /mnt/zeit

Write barriers disabled, write cache enabled:

shambhala:~> hdparm -W /dev/sda

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=5376750, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=2625, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

shambhala:~> mount -o nobarrier /dev/sda6 /mnt/zeit shambhala:~> cd /mnt/zeit
shambhala:/mnt/zeit> sync ; time
tar -xf /home/martin/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2; time sync
tar -xf /home/martin/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 26,77s user 3,20s system 67% cpu 44,236 total
sync 0,00s user 0,17s system 5% cpu 3,224 total shambhala:/mnt/zeit> sync ; time rm -rf linux-2.6.27 ; time sync rm -rf linux-2.6.27 0,05s user 3,20s system 43% cpu 7,442 total sync 0,00s user 0,14s system 44% cpu 0,309 total shambhala:/mnt/zeit> cd
shambhala:~> umount /mnt/zeit

Write barriers and write cache disabled:

shambhala:~> hdparm -W0 /dev/sda

/dev/sda:
setting drive write-caching to 0 (off) write-caching = 0 (off)
shambhala:~> hdparm -I /dev/sda | grep cache
Write cache
shambhala:~> mkfs.xfs -f -L xfs -l lazy-count=1 /dev/sda6
meta-data=/dev/sda6 isize=256 agcount=4, agsize=1344188
blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=5376750, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=2625, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

shambhala:~> mount -o nobarrier /dev/sda6 /mnt/zeit shambhala:~> cd /mnt/zeit
shambhala:/mnt/zeit> sync ; time
tar -xf /home/martin/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2; time sync
tar -xf /home/martin/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 35,22s user 3,89s system 53% cpu 1:12,71 total
sync 0,00s user 0,17s system 7% cpu 2,355 total shambhala:/mnt/zeit> sync ; time rm -rf linux-2.6.27 ; time sync rm -rf linux-2.6.27 0,07s user 2,82s system 12% cpu 23,391 total sync 0,00s user 0,14s system 44% cpu 0,329 total shambhala:/mnt/zeit> cd
shambhala:~> umount /mnt/zeit

shambhala:~> date
Mo 8. Dez 21:24:42 CET 2008

[1] http://oss.sgi.com/archives/xfs/2008-12/msg00219.html
[2] http://oss.sgi.com/archives/xfs/2008-12/msg00161.html or
http://lkml.org/lkml/2008/12/4/169

Ciao,

Signature · 2 lines

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Martin Steigerwald

author

2008-12-08 21:43 UTC #1 lore raw

Am Montag 08 Dezember 2008 schrieb Martin Steigerwald:

Quoted 9 lines

Hi!
I got curious about the recent discussions about the write barrier feature[1][2].
Thus I did my own benchmark this evening. Since I use XFS and tested with XFS for now only. Write barrier + write cache, no barrier + write cache, no write barrier + no write cache. I just did tar -xf linux-2.6.27.tar.bz2 and rm -rf linux-2.6.27.

I disabled write cache and barriers for my XFS filesystems on my TP 42 shambhala. During that I did further tests. This time I also include IO scheduler settings. They are default and have been the same during my previous tests.

Write cache has about 13 seconds benefit on the barrier enabled XFS for the tar -xf linux-2.6.27.tar.bz2 workload but no benefit at all for the rm -rf linux-2.6.27 workload!

And again nobarrier and no cache wins.

On a heavily used /home-XFS - I know it should have more free space:

martin@shambhala:~ -> LANG=C df -hT /home/

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda5      xfs    112G  107G  5.1G  96% /home

It has been grown two times - since 6 AGs instead of the default of 4:

shambhala:~> xfs_info /home
meta-data=/dev/sda5 isize=256 agcount=6, agsize=4883256
blks

         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=29299536, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

IO scheduler settings:

martin@shambhala:~> cat /sys/block/sda/queue/scheduler noop anticipatory deadline [cfq] martin@shambhala:~> grep "" /sys/block/sda/queue/iosched/* /sys/block/sda/queue/iosched/back_seek_max:16384 /sys/block/sda/queue/iosched/back_seek_penalty:2 /sys/block/sda/queue/iosched/fifo_expire_async:250 /sys/block/sda/queue/iosched/fifo_expire_sync:123 /sys/block/sda/queue/iosched/quantum:4 /sys/block/sda/queue/iosched/slice_async:40 /sys/block/sda/queue/iosched/slice_async_rq:2 /sys/block/sda/queue/iosched/slice_idle:6 /sys/block/sda/queue/iosched/slice_sync:100

Write barriers and no write cache:

shambhala:~> hdparm -W /dev/sda

/dev/sda:
write-caching = 0 (off)

shambhala:> cat /proc/mounts | egrep "(/ |home)"
rootfs / rootfs rw 0 0 /dev/disk/by-uuid/7fcd9766-bf1a-426a-8a07-2c3e0c510898 / xfs rw,relatime,attr2,noquota 0 0
/dev/sda5 /home xfs rw,relatime,attr2,logbufs=8,logbsize=256k,noquota 0 0 martin@shambhala:~/Zeit> sync; time tar -xf ~/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 ; time sync
tar -xf ~/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 36,71s user 4,22s system 36% cpu 1:51,58 total
sync 0,00s user 0,05s system 0% cpu 12,689 total martin@shambhala:~/Zeit> sync; time rm -rf linux-2.6.27 ; time sync rm -rf linux-2.6.27 0,06s user 3,90s system 17% cpu 22,906 total sync 0,00s user 0,01s system 6% cpu 0,103 total

Write barriers and write cache:

shambhala:~> hdparm -W1 /dev/sda

/dev/sda:
setting drive write-caching to 1 (on)
write-caching = 1 (on)

martin@shambhala:~/Zeit> sync; time tar -xf ~/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 ; time sync
tar -xf ~/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 34,38s user 3,91s system 38% cpu 1:38,84 total
sync 0,00s user 0,04s system 0% cpu 10,493 total martin@shambhala:~/Zeit> sync; time rm -rf linux-2.6.27 ; time sync rm -rf linux-2.6.27 0,07s user 3,98s system 17% cpu 23,511 total sync 0,00s user 0,01s system 5% cpu 0,126 total

No write barriers and no write cache:

shambhala:> vim fstab
shambhala:> mount -o remount /
shambhala:> mount -o remount /home
shambhala:> cat /proc/mounts | egrep "(/ |home)" rootfs / rootfs rw 0 0 /dev/disk/by-uuid/7fcd9766-bf1a-426a-8a07-2c3e0c510898 / xfs rw,relatime,attr2,nobarrier,noquota 0 0
/dev/sda5 /home xfs rw,relatime,attr2,nobarrier,logbufs=8,logbsize=256k,noquota 0 0

martin@shambhala:~/Zeit> sync; time tar -xf ~/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 ; time sync
tar -xf ~/Linux/Kernel/Mainline/linux-2.6.27.tar.bz2 30,36s user 3,31s system 48% cpu 1:08,94 total
sync 0,00s user 0,08s system 0% cpu 17,462 total martin@shambhala:~/Zeit> sync; time rm -rf linux-2.6.27 ; time sync rm -rf linux-2.6.27 0,07s user 3,87s system 20% cpu 19,172 total sync 0,00s user 0,01s system 4% cpu 0,142 total

martin@shambhala:~> date
Mo 8. Dez 22:38:06 CET 2008

Ciao,

Signature · 2 lines

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7