Skip to contentSkip to conversation

ext4 is faster with LVM than without, and other filesystem benchmarks

DISCUSSION Sami Liedes 4 participants 4 messages May 7, 2010

lore
Share & tools
lore
#0 lore raw

[Please Cc: the answers to me, I'm not subscribed.]

[As this concerns both filesystems and device-mapper, and I'm hesitant to crosspost to kernel lists, I'm posting to lkml. Feel free to continue the discussion on more appropriate lists.]

Hello lkml,

While moving my backups from one partition to another, I accidentally generated a tar archive which seems to stress filesystems in an interesting way when extracted. It's a rather narrow but real-world benchmark. The contents are those of a backup hierarchy generated by BackupPC[1]. The size of the tar (uncompressed) is about 492 gibibytes. The first 490 GiB are extracted quickly (>100 MiB/s), the extraction of the last 2 GiB takes hours.

I've run some benchmarks on ext4 and btrfs, and very limited benchmarks on xfs, jfs, nilfs2 and zfs-fuse, all mounted with noatime but otherwise default options. The major focus was on ext4 behavior after establishing the ballpark figures of how well each filesystem performs compared to others. I'm posting the results in the hope that they will be of interest at least to filesystem and device-mapper developers.

The results are at

http://www.hut.fi/~sliedes/fsspeed/fsspeed.html

and also summarized here.

I ran the benchmarks on a raw disk partition (/dev/sdd1), on LVM device-mapper linear mapping to the same partition, and on LVM dm linear mapping and dm-crypt on top of it. The disk is not simultaneously used for anything else.

Basically the directory hierarchy consists of

* directory backuppc/cpool - a pool of compressed backed up files
- The individual files are of the form

    backuppc/cpool/f/1/d/f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
  - There are some 456000 files in the pool, all hashed by their
    SHA256 sum

- These take the first about 490 gibibytes of the archive

* directory backuppc/pc with individual backups of a few machines
- Essentially contains the root filesystems of quite normal desktop Linux machines.
- All backed up files are hardlinks to the pool.
- Files with size 0 are not hardlinked but represented as themselves.
- Contains some 10.4M hard links to files in the pool, interspersed with some 84300 empty files and very few regular files

The benchmarks were all I/O-bound on the speed of the target filesystem and partition. This was achieved by making a copy of the tar that has all the file contents zeroed and using a fast compressor (lzop) so the decompression of the .tar.lzop was still easily target-disk-bound.

All the benchmarks were run on a Core 2 Quad Q6600 2.40 GHz computer with 8 GiB of RAM and vanilla kernel 2.6.33.2. The computer was not a dedicated machine, however load was otherwise fairly light and never on the /dev/sdd disk. I tend to think this as a benefit, since this way the results represent performance as it varies under light load, if the load has any effect. /dev/sdd is a 1500 gigabyte Seagate Barracuda 7200.11 disk and /dev/sdd1 is the only partition on it.

I benchmarked two different things: The behavior
of extracting the tar, and after that (without re/unmounting in between) rm -rf'ing the resulting directory structure.

I think the most surprising result of the benchmarks is that ext4 seems to be significantly faster under LVM than on the raw /dev/sdd1 partition. Without LVM the variance in run times is much larger.

Here are the numbers for ext4 with and without LVM. For the raw results and some nice graphs, see [2]. All times are in seconds.

Average is the average time taken, min the minimum and max the maximum. stddev is the sample standard deviation of the sample and stddev% as percentage from average.

If we assume that the times are normally distributed (which they possibly aren't exactly), the real mean/average lies between "avg min" and "avg max" with 95% confidence, the distance from average being the number in the "avg ± 95%" column.

Caveat: I'm not much of a statistician, so I might have made some mistakes. See [2] for the calculations.

------------------------------------------------------------
ext4, /dev/sdd1 (no lvm), noatime, 9 samples:

extract rm
average 20638 17518.11

min		     18567	    14913
max		     25670	    21196

stddev 2137.86 2060.71
stddev % 10.359% 11.763%
avg ± 95% 1440.96 1388.96
avg ± % 6.982% 7.929%
avg min 19197.49 16129.15
avg max(*) 22079.40 18907.07
------------------------------------------------------------
ext4, LVM, dm linear mapping on /dev/sdd1, 7 samples:

# dmsetup table
[...]
rootvg-testdisk: 0 2720571392 linear 8:49 384

extract rm
average 19475.7 13634.6

min		     18006	    12036
max		     20247	    14367
stddev		     738.341	    763.391
stddev %	     3.791%	    5.599%
avg ± 95%	     570.136	    589.479
avg ± %		     2.927%	    4.323%
avg min		     18905.6	    13045.1
avg max		     20045.9	    14224.0

------------------------------------------------------------
ext4, LVM and dm-crypt, dm linear mapping on /dev/sdd1, 5 samples:

# dmsetup table
rootvg-testdisk: 0 2720571392 linear 8:49 384
rootvg-testdisk_crypt: 0 2720569336 crypt aes-cbc-essiv:sha256 0000000000000000000000000000000000000000000000000000000000000000 0 253:3 2056

extract rm
average 25541.4 15690.4

min		     22618	    13766
max		     28550	    18018
stddev		     2315.48	    1687.58
stddev %	     9.066%	    10.756%
avg ± 95%	     2159.19	    1573.67
avg ± %		     8.454%	    10.030%
avg min		     23382.2	    14116.7
avg max		     27700.6	    17264.1

------------------------------------------------------------

With dm-crypt, the extraction is CPU bound on the computer, so the extraction benchmark is not so interesting there. However what is interesting is the rm benchmark; it seems that ext4 under LVM is consistently faster than ext4 on the raw partition (no LVM).

So far I haven't seen the same effect on btrfs, however my btrfs sample size is quite small (see [2] for the measurements).

btrfs performs significantly better in terms of run time than ext4. However in a sense it fails the test: A number of the hard links are actually not extracted, since btrfs places quite heavy restrictions on the number of hard links (kernel bugzilla #15762). The number of hard links that failed to extract is so low however that I do not believe it has an effect on the times. Here are the numbers for btrfs:

------------------------------------------------------------
btrfs, /dev/sdd1 (no lvm), noatime, 5 samples:

extract rm
average 13514 8805.75

min		     12424	    8024
max		     15202	    9489
stddev		     1186.96	    623.663
stddev %	     8.783%	    7.082%
avg ± 95%	     1262.56	    663.39
avg ± %		     9.343%	    7.534%
avg min		     12251.4	    8142.36
avg max		     14776.6	    9469.14

------------------------------------------------------------

Also a significant difference in rm performance to ext4, if we plot the number of files removed so far as a function of time (see [2]), is that while ext4 offers a fairly steady performance in terms of files removed per second - it's mostly a straight line - btrfs clearly accelerates towards the end. On ext4 a single unlink() syscall may pause for at most tens of seconds, and even that's rare, while on btrfs the process frequently stalls for as much as minutes, blocked in unlink(). So while btrfs is faster in terms of total time, it can create huge latencies for the calling process.

One thing that I have figured out affects the rm time on ext4 is the order in which rm happens to remove the directories, i.e. pool first or the backuppc/pc directory first. Removing the pool first seems to cause the remaining operation to be faster. This is visible in the plots of number of removed files so far as function of time as a steep short curve either in the beginning of the operation (pool removed first) or the end (pool removed last). [2]

On XFS, extracting the archive took more than 16 hours (58548 seconds; only one sample), so XFS performs rather poorly in this I assume quite pathological case. On JFS, extraction took more than 19 hours (69365 seconds) and rm took 52702 seconds. On zfs-fuse I did not wait for the extraction to finish, but a rough guesstimate is that it would have taken slightly over a week.

The one filesystem that really beat every other in this particular benchmark was nilfs2. On it, the metadata-heavy hard link extraction phase was only slightly slower than the data-heavy pool extraction phase. While on ext4 typically the data-heavy part took maybe 1.3 hours and the rest took 3-4 hours, on nilfs2 the data-heavy part took 2 hours and the metadata-heavy typically pathological part only 15 minutes. Also rm only took 40 minutes (2419 seconds; only one sample here too).

***Like btrfs, nilfs2 too failed to support enough hard links to contain the directory structure***. Again most files were correctly extracted, so I don't believe that had a major impact on efficiency.

See the graphs, data and further analysis at [2].

Sami

[1] http://backuppc.sourceforge.net/
[2] http://www.hut.fi/~sliedes/fsspeed/fsspeed.html

#1 lore raw
Quoted 3 lines · In article <[email protected]> you wrote:

I think the most surprising result of the benchmarks is that ext4 seems to be significantly faster under LVM than on the raw /dev/sdd1 partition. Without LVM the variance in run times is much larger.

Sounds like the syncs/barriers never reach the disk with LVM. What mount journal/sync options did you use? default? Is dmesg telling you that barriers have been turned off for the lvm case? IF yo uwant it fast and dirty try nobarrier,data=writeback :)

Greetings
Bernd

#2 lore raw
Quoted 9 lines · Bernd Eckenfels
Quoted 3 lines · In article <[email protected]> you wrote:

I think the most surprising result of the benchmarks is that ext4 seems to be significantly faster under LVM than on the raw /dev/sdd1 partition. Without LVM the variance in run times is much larger.

Sounds like the syncs/barriers never reach the disk with LVM. What mount journal/sync options did you use? default? Is dmesg telling you that barriers have been turned off for the lvm case? IF yo uwant it fast and dirty try nobarrier,data=writeback :)

LVM barriers should be working in that kernel, though... still, comparing mount -o nobarrier on each might yield a clue there.

-Eric

#3 lore raw
In reply to Sami Liedes
Quoted 22 lines · Sami Liedes

Basically the directory hierarchy consists of

* directory backuppc/cpool - a pool of compressed backed up files
- The individual files are of the form

    backuppc/cpool/f/1/d/f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
  - There are some 456000 files in the pool, all hashed by their
    SHA256 sum

- These take the first about 490 gibibytes of the archive

* directory backuppc/pc with individual backups of a few machines
- Essentially contains the root filesystems of quite normal desktop Linux machines.
- All backed up files are hardlinks to the pool.
- Files with size 0 are not hardlinked but represented as themselves.
- Contains some 10.4M hard links to files in the pool, interspersed with some 84300 empty files and very few regular files

The benchmarks were all I/O-bound on the speed of the target filesystem and partition. This was achieved by making a copy of the tar that has all the file contents zeroed and using a fast compressor (lzop) so the decompression of the .tar.lzop was still easily target-disk-bound.

[...]

Quoted 3 lines

On XFS, extracting the archive took more than 16 hours (58548 seconds; only one sample), so XFS performs rather poorly in this I assume quite pathological case.

Not pathological, but XFS is generally slower at creating and removing large numbers of files compared to ext and btrfs until you get to parallel workloads and expensive storage hardware. However, just adding the "logbsize=262144" mount option should speed it up significantly for this workload.

FWIW, how big is the compressed tarball you are extracting? If it's not large, can you make it available somewhere? I've current got a set of pending modifications to XFS(*) that should help this workload and this looks like a good load for testing...

Cheers,

Dave.

(*) http://oss.sgi.com/archives/xfs/2010-05/msg00060.html

Signature · 2 lines
Dave Chinner
[email protected]