abysmal rm performance?
I'm using 3.10 with a btrfs filesystem with RAID-1 (on two drives), with extended inode refs and skinny metadata extent refs enabled (-r and -x options in btrfstune).
Server has 32 GB RAM.
Filesystem is mounted with noatime,compress-force=zlib mount options.
btrfs performs really, really poor when removing files.
Some examples - removing files for 10 seconds, repeated 10 times in a row.
Each time, we measure how many files we removed, and amount of memory we have to write to disk after rm operation ("Dirty"
from /proc/meminfo):
TIMEOUT=10s
sync
timeout $TIMEOUT rm -rfv trash_dir/ &>/tmp/rmout.log wc -l /tmp/rmout.log
grep Dirty /proc/meminfo
Removed files: 4319 Dirty: 211956 kB
Removed files: 3392 Dirty: 190764 kB
Removed files: 4011 Dirty: 174636 kB
Removed files: 5197 Dirty: 191500 kB
Removed files: 6395 Dirty: 202532 kB
Removed files: 4613 Dirty: 354764 kB
Removed files: 5469 Dirty: 170664 kB
Removed files: 4654 Dirty: 170876 kB
Removed files: 2245 Dirty: 152108 kB
Removed files: 2214 Dirty: 149848 kB
Compare it to ext4 - note "Dirty" is an order of magnitude lower for ext4:
Removed files: 7346 Dirty: 4896 kB
Removed files: 11770 Dirty: 3536 kB
Removed files: 4266 Dirty: 80 kB
Removed files: 7541 Dirty: 4164 kB
Removed files: 8046 Dirty: 5428 kB
Removed files: 9630 Dirty: 5884 kB
Removed files: 14276 Dirty: 8384 kB
Removed files: 34234 Dirty: 10968 kB
Removed files: 10594 Dirty: 4348 kB
Removed files: 22672 Dirty: 4164 kB
File removal is actually quite fast until we reach around 350000 kB in "Dirty" (this is with 32 GB RAM). Then, it's super slow. Let's see what happens if we remove the files for 1 minute (above was for just 10 secs):
btrfs:
Removed files: 18360 Dirty: 98276 kB
Removed files: 9913 Dirty: 60664 kB
Removed files: 10973 Dirty: 62284 kB
Removed files: 16606 Dirty: 275156 kB
Removed files: 13002 Dirty: 165844 kB
Removed files: 8349 Dirty: 178448 kB
Removed files: 20316 Dirty: 394912 kB
Removed files: 19109 Dirty: 321252 kB
Removed files: 22738 Dirty: 277964 kB
Removed files: 15288 Dirty: 41400 kB
ext4:
Removed files: 91714 Dirty: 7060 kB
Removed files: 79574 Dirty: 400 kB
Removed files: 105167 Dirty: 5384 kB
Removed files: 37123 Dirty: 25572 kB
Removed files: 94048 Dirty: 13708 kB
Removed files: 149079 Dirty: 48592 kB
Removed files: 136770 Dirty: 528 kB
Removed files: 169513 Dirty: 21024 kB
Removed files: 171877 Dirty: 1936 kB
Removed files: 95442 Dirty: 7780 kB
So it looks like removing files with btrfs needs much more metadata updates?
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Tomasz Chmielewski posted on Sat, 20 Jul 2013 13:37:26 +0800 as excerpted:
Quoted 2 lines
So it looks like removing files with btrfs [as opposed to ext4] needs much more metadata updates?
You /really/ need to read up on the btrfs wiki.
The short answer is yes, btrfs does a LOT more metadata processing due to the checksumming it does by default. (Consider that it must have all the metadata from a leaf available in ordered to rechecksum it when one file's metadata from that leaf gets deleted.) Additionally, btrfs keeps two copies of metadata by default, in raid1 mode if there's multiple devices (btrfs raid1), DUP mode if not (other forms of raid, which would appear to btrfs as a single device).
Then there's the whole problem that you didn't provide nearly enough information about your test to tell what it was actually comparing. What sort of raid1, btrfs/md/dm/hardware/what, and if btrfs raid1, was that for both data and metadata or just one of the two and what was the other one if they weren't both raid1? And if you were testing btrfs raid1, what did you do with the ext4 test to try to make it comparable since ext4 doesn't have a native raid1 mode, or was it on a single device?
So... read up on the wiki a bit, then come back with questions you have that aren't answered there. (I certainly had some I didn't see directly answered there when I first started with btrfs.)
https://btrfs.wiki.kernel.org/
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 3 lines
So... read up on the wiki a bit, then come back with questions you have that aren't answered there. (I certainly had some I didn't see directly answered there when I first started with btrfs.)
I guess the original email was more ment as a bug-report than a question, as the question was more like a "can it really be that slow".
The wiki most likely won't help explaining/solving the high metadata overhead either...
Regards