Oddly slow read performance with near-full largish FS

Hi,

I've been running btrfs for various filesystems for a few years now, and have recently run into problems with a large filesystem becoming *really* slow for basic reading. None of the debugging/testing suggestions I've come across in the wiki or in the mailing list archives seems to have helped.

Background: this particular filesystem holds backups for various other machines on the network, a mix of rdiff-backup data (so lots of small files) and rsync copies of larger files (everything from ~5MB data files to ~60GB VM HD images). There's roughly 16TB of data in this filesystem (the filesystem is ~17TB). The btrfs filesystem is a simple single volume, no snapshots, multiple devices, or anything like that. It's an LVM logical volume on top of dmcrypt on top of an mdadm RAID set (8 disks in RAID 6).

The performance: trying to copy the data off this filesystem to another (non-btrfs) filesystem with rsync or just cp was taking aaaages - I found one suggestion that it could be because updating the atimes required a COW of the metadata in btrfs, so I mounted the filesystem noatime, but this doesn't appear to have made any difference. The speeds I'm seeing (with iotop) fluctuate a lot. They spend most of the time in the range of 1-3 MB/s, with large periods of time where no IO seems to happen at all, and occasional short spikes to ~25-30 MB/s. System load seems to sit around 10-12 (with only 2 processes reported as running, everything else sleeping) while this happens. The server is doing nothing other than this copy at the time. The only processes using any noticable CPU are rsync (source and destination processes, around 3% CPU each, plus an md0:raid6 process around 2-3%), and a handful of "kworker" processes, perhaps one per CPU (there are 8 physical cores in the server, plus hyperthreading).

Other filesystems on the same physical disks have no trouble exceeding 100MB/s reads. The machine is not swapping (16GB RAM, ~8GB swap with 0 swap used).

Is there something obvious I'm missing here? Is there a reason I can only average ~3MB/s reads from a btrfs filesystem?

kernel is x86_64 linux-stable 3.17.6. btrfs-progs is v3.17.3-3-g8cb0438. Output of the various info commands is:

  $ sudo btrfs fi df /media/backup/
  Data, single: total=16.24TiB, used=15.73TiB
  System, DUP: total=8.00MiB, used=1.75MiB
  System, single: total=4.00MiB, used=0.00
  Metadata, DUP: total=35.50GiB, used=34.05GiB
  Metadata, single: total=8.00MiB, used=0.00
  unknown, single: total=512.00MiB, used=0.00

  $ btrfs --version
  Btrfs v3.17.3-3-g8cb0438

  $ sudo btrfs fi show

Label: 'backup' uuid: c18dfd04-d931-4269-b999-e94df3b1918c
Total devices 1 FS bytes used 15.76TiB
devid 1 size 16.37TiB used 16.31TiB path /dev/mapper/vg-backup

Thanks in advance for any suggestions.

Charles

-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

Hi,

Sorry for late reply. Let me ask some questions.

Q1. You mean your Btrfs file system exists on the top of
the following deep layers?

+---------------+
|Btrfs(single)  |
+---------------+
|LVM(non RAID?) |
+---------------+
|dmcrypt        |
+---------------+
|mdadm RAID set |
+---------------+

# Unfortunately, I don't know how Btrfs works in conjunction #with such a deep layers.

Q2. If Q1 is true, is it possible to reduce that layers as follows?

+-----------+
|Btrfs(*1)  |
+-----------+
|dmcrypt    |
+-----------+

It's because there are too many layers and these have the same/similar features and heavy layered file system tends to cause more trouble than thinner layered ones regardless of file system type.

*1) Currently I don't recommend you to use RAID56 of Btrfs.
So, if RAID6 is mandatory, mdadm RAID6 is also necessary.

Q3. They are also consist of the following layers?

+---------------+
|XFS/ext4       |
+---------------+
|LVM(non RAID?) |
+---------------+
|dmcrypt        |
+---------------+
|mdadm RAID set |
+---------------+

Q4. Are other filesystems also near-full?

Q5. Is there any error/warning message about
Btrfs/LVM/dmcrypt/mdadm/hardwares?

Thanks,
Satoru

Sure - thanks for taking an interest.

Yes, precisely. mdadm is used to make a large RAID6 device, which is encrypted with LUKS, on top of which is layered LVM (for ease of management), and the btrfs filesystem sits on that.

I don't see how I could do that - I simply have far too much data for a single disk (not to mention I don't want to risk loss of data from a single disk failing). This filesystem has 16.x TB of data in it at present.

This configuration is one I've been using for many years. It's only recently that I've noticed it being particularly slow with btrfs -- I don't know if that's because the filesystem has filled up past some critical point, or due to something else entirely. That's why I'm trying to figure this out.

Yes, exactly. That's why I use mdadm.

[...]

Yes, exactly the same configuration. The fact that I don't see any speed problems with other filesystems (even in the same LVM volume group) leads me in the direction of suspecting something to do with btrfs.

No, not particularly. Now, the btrfs volume in question isn't exactly close to full - there's more than 500 GB free. It's just *relatively* full.

No, no errors or warnings in logs related to the disks, LVM, or btrfs. I have historically, with previous kernels, gotten the "task blocked for more than 120 seconds" warnings fairly often, but I haven't seen those lately.

Is there any other info I can collect on this that would help?

Thanks,

Charles

-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

Charles Cazabon posted on Fri, 19 Dec 2014 10:58:49 -0600 as excerpted:

Not recommending at this point, just saying these are options...

Btrfs raid56 mode should, I believe, be pretty close to done with the latest patches. That would be 3.19, however, which isn't out yet of course.

There's also raid10, if you have enough devices or little enough data to do it. That's much more mature than raid56 mode and should be about as mature and stable as btrfs in single-device mode, which is what you are using now. But it'll require more devices than a raid56 would...

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

And also with such large storage stacks with big drives, when they fail (note I use when not if) it takes a long time to restore. So if you have the ability to break them up and use something like GlusterFS to distribute it, it helps to mitigate this as well as other kinds of failures like power supply, logic board, controllers, and with georep even the entire local site.

This is not meant to indicate the current layout is wrong. Just that there are other possibilities to achieve the desired up-time and data safety.

Chris Murphy

Putting the encryption above the raid is a _huge_ win that he'd lose.

I've used this same layering before (though not with btrfs).

So if you write a sector in this order only one encryption event (e.g. "encrypt this sector") has to take place no matter what raid level is in place.

If you put the encryption below the raid, then a write or one sector on a non-degraded RAID5 requires four encryption events (two decrypts, one for the parity and one for the sector being overwritten; followed by two encryptions on the same results).

In degraded conditions the profile is much worse.

If encryptions and RAID > 0 is in use, he's better off with what he's got in terms of CPU and scheduling.

Totally spit-balling ideas here (e.g. no suggestion as to which one to try first etc, just typing them as they come to me):

Have you tried increasing the number of stripe buffers for the filesystem? If you've gotten things spread way out you might be thrashing your stripe cache. (see /sys/block/md(number here)/md/stripe_cache_size).

Have you taken SMART (smartmotools etc) to these disks to see if any of them are reporting any sort of incipient failure conditions? If one or more drives is reporting recoverable read errors it might just be clogging you up.

Try experimentally mounting the filesystem read-only and dong some read tests. This elimination of all possible write sources will tell you things. In particular if all your reads just start breezing through then you know something in the write path is "iffy". One thing that comes to mind is that anything accessing the drive with a barrier-style operation (wait for verification of data sync all the way to disk) would have to pass all the way down through the encryption layer which could be having a multiplier effect. (you know, lots of very short delays making a large net delay).

Have you changed any hardware lately in a way that could de-optimize your interrupt handling.

I have a vague recollection that somewhere in the last month and a half or so there was a patch here (or in the kernel changelogs) about an extra put operation (or something) that would cause a worker thread to roll over to -1, then spin back down to zero before work could proceed. I know, could I _be_ more vague? Right? Try switching to kernel 3.18.1 to see if the issue just goes away. (Honestly this one's just been scratching at my brain since I started writing this reply and I just _can't_ remember the reference for it... dangit...)

When was the last time you did any of the maintenance things (like balance or defrag)? Not that I'd want to sit through 15Tb of that sort of thing, but I'm curious about the maintenance history.

Does the read performance fall off with uptime? E.g. is it "okay" right after a system boot and then start to fall off as uptime (and activity) increases? I _imagine_ that if your filesystem huge and your server is modest by comparison in terms of ram, cache pinning and fragmentation can start becoming a real problem. What else besides marshaling this filesystem is this system used for?

Have you tried segregating some of your system memory for to make sure that you aren't actually having application performance issues? I've had some luck with kernelcore= and moveablecore= (particularly moveablecore=) kernel command line options when dealing with IO induced fragmentation. On problematic systems I'll try classifying at least 1/4 of the system ram as movablecore. (e.g. on my 8GiB laptop were I do some of my experimental work, I have moveablecore=2G on the command line). Any pages that get locked into memory will be moved out of the movable-only memory first. This can have a profound (usually positive) effect on applications that want to spread out in memory. If you are running anything that likes large swaths of memory then this can help a lot. Particularly if you are also running programs that traverse large swaths of disk. Some programs (rsync of large files etc may be such a program) can do "much better" if you've done this. (BUT DON'T OVERDO IT, enough is good but too much is very bad. 8-) ).

ASIDE: Anything that uses hugepages, transparent or explicit, in any serious number has a tendency to antagonize the system cache (and vice-versa). It's a silent fight of the cache-pressure sort. When you explicitly declare an amount of ram for moveable pages only, the disk cache will not grow into that space. so moveablecore=3G creates 3GiB of space where only unlocked pages (malloced heap, stack, etc; basically only things that can get moved -- particularly swapped -- will go in that space.) The practical effect is that certain kinds of pressures will never compete. So broad-format disk I/O (e.g. using find etc) will tend to be on one side of the barrier while video playback buffers and virtual machine's ram regions are on the other. The broad and deep filesystem you describe could be thwarting your program's attempt to access it. That is, the rsync's need to load a large number of inodes could be starving rsync for memory (etc). Keeping the disk cache out of your program's space at least in part could prevent some very "interesting" contention models from ruining your day.

Or it could just make things worse.

So it's worth a try but it's not gospel. 8-)

Hi, Robert,

Thanks for the response. Many of the things you mentioned I have tried, but for completeness:

Yes. The disks are actually connected to a proper hardware RAID controller that does SMART monitoring of all the disks, although I don't use the RAID features of the controller. By using mdadm, if the controller fails I can slap the disks in another machine, or a different controller into this one, and still have it work without needing to worry about getting a replacement for this particular model of controller.

There are no errors or warnings from SMART for the disks.

Actually, I'd already done that before I mailed the list. It made no difference to the symptoms.

No.

I tend to track linux-stable pretty closely (as that seems to be recommended for btrfs use), so I already switched to 3.18.1 as soon as it came out. That made no difference to the symptoms either.

I don't generally do those at all. I was under the impression that balance would not apply in my case as btrfs is on a single logical device, but I see that I was wrong in that impression. Is this something that is recommended on a regular basis? Most of the advice I've read regarding them is that it's no longer necessary unless there is a particular problem that these will fix...

No. I see these problems right from boot.

This particular server is only used for holding backups of other machines, nothing else. It has far more CPU and memory (2x quad-core Xeon plus hyperthreading, 16GB RAM) than it needs for this task. So when I say the machine is doing nothing other than this copy/rsync I'm currently running, that's practically the literal truth - there are the normal system processes and my ssh/shell running and that's about it.

The system isn't running out of memory; as I say, about the only userspace processes running are ssh, my shell, and rsync.

However, your first suggestion caused me to slap myself:

This I had totally forgotten. When I bump up the stripe cache size, it *seems* (so far, at least) to eliminate the slowest performance I'm seeing - specifically, the periods I've been seeing where no I/O at all seems to happen, plus the long runs of 1-3MB/s. The copy is now staying pretty much in the 22-27MB/s range.

That's not as fast as the hardware is capable of - as I say, with other filesystems on the same hardware, I can easily see 100+MB/s - but it's much better than it was.

Is this remaining difference (25 vs 100+ MB/s) simply due to btrfs not being tuned for performance yet, or is there something else I'm probably overlooking?

Thanks,

Charles

-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

Do make sure you are regularly running the long "offline" test. [offline is a bad name, what it really should be called is the long idle-interval test. sigh] about once a week. Otherwise SMART is just going to tell you the disk just died when it dies.

I'm not saying this is relevant to the current circumstance. But since you didn't mention a testing schedule I figured it bared a mention

The thing with "movablecore=" will not lead to an "out of memory" condition or not, its a question of cache and buffer evictions.

I figured that you'd have said something about actual out of memory errors.

But here's the thing.

Once storage pressure gets "high enough" the system will start forgetting things intermittently to make room for other things. One of the things it will "forget" is pages of code from running programs. The other thing it can "forget" is dirent (directory entries) relevant to ongoing activity.

The real killer can involve "swappiness" (e.g. /proc/sys/vm/swapiness :: the tendency of the system to drop pages of program code, do not adjust this till you understand it fully) and overall page fault rates on the system. You'll start geting evictions long before you start using _any_ swap file space.

So if your effective throughput is low, the first thing to really look at is if your page fault rates are rising. Variations of sar, ps, and top may be able to tell you about the current system and/or per-process page fault rates. You'll have to compare your distro's tool set to the procedures you can find online.

It's a little pernicious because it's a silent performance drain. There are no system messages to tell you "uh, hey dude, I'm doing a lot of reclaims lately and even going back to disk for pages of this program you really like". You just have to know how to look in that area.

I find BTRFS can be a little slow on my laptop, but I blame memory pressure evicting important structures somewhat system wide. Which is part of why I did the moveablecore= parametric tuning. I don't think there is anything that will pack the locality of the various trees, so you can end up needing bits of things from all over your disk in order to sequentially resolve a large directory and compute the running checksums for rsync (etc.).

Simple rule of thumb, if "wait for I/O time" has started to rise you've got some odd memory pressure that's sending you to idle land. It's not hard-and-fast as a rule, but since you've said that your CPU load (wich I'm taking to be the user+system time) is staying low you are likely waiting for something.

Hi, Robert,

My performance issues with btrfs are more-or-less resolved now -- the performance under btrfs still seems quite variable compared to other filesystems -- my rsync speed is now varying between 40MB and ~90MB/s, with occasional intervals where it drops further, down into the 10-20MB/s range. Still no disk errors or SMART warnings that would indicate that problem is at the hardware level.

Ok, I'll do that.

Ya, I'm aware of how limited/useful the SMART diagnostics are. I'm also paranoid enough to be using RAID 6...

I'm fairly certain memory isn't the issue here. For what it's worth:

%Cpu(s): 2.1 us, 19.4 sy, 0.0 ni, 78.0 id, 0.2 wa, 0.3 hi, 0.0 si, 0.0 st

KiB Mem:  16469880 total, 16301252 used,   168628 free,      720 buffers
KiB Swap:  7811068 total,        0 used,  7811068 free, 15146580 cached

Swappiness I've left at the default of 60, but I'm not seeing swapping going on regardless.

I found the cause of this. Stupidly enough, there was a bwlimit set up in a shell alias for rsync.

So btrfs is not nearly as slow as I was seeing. It's still slower than reading from an ext4 or XFS filesystem on these disks, but the absolute level of read speed seems reasonable enough given that btrfs has not been under heavy performance tuning to date. My only remaining concern would be the variability I still see in the read speed.

Charles

-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

Swappiness has nothing to do with swapping.

You have very little free memory.

Here is how a linux system runs a program. When exec() is called the current process memory is wiped (dropped, forgotten, whatever) [except for a few things like the open file discriptor table]. Then the executable is opened and mmap() is called to map the text portions of that executable into memory. This does not involve any particular reading of that file. The dynamic linker also selectively mmap()s the needed libraries. So you end up with something that looks like this:

[+.....................]

Most of the program is not actually in memory. (the "." parts), and some minimum part is in memory (the "+" part).

As you use the program more of it will work its way into memory.

[+++.....+.+....++....]

swappiness controls the likelihood that memory pressure will cause parts that have been read in to be "forgotten" based on the idea that it can be read again later if needed.

[+++.....+.+....++....]
[+.......+......+.....]

This is called "demand paging", and because linux uses ELF (extensible link format) and all programs run in a uniform memory map, program text _never_ needs to be written to swap space. Windows DLL/EXE has to "relocate" the code, e.g. re-write it to make it runable. So on windows code text has to be paged to swap.

So what is "swapping" and swap space for?

Well next to the code is the data.

[+++.....+.+....++....] {.............}

As it gets written to, it cannot just be forgotten because the program needs that data or it wouldn't have written it.

[+++.....+.+....++....] {..******.*...}

So if the system needs to reclaim the memory used by that kind of data it sends it to the swap space.

[+++.....+.+....++....] {..******.*...}
^^^^ swapping vvvv
[+++.....+.+....++....] {..**...*.*...}

Swappiness is how you tel the system that you want to keep the code ("+") in memory in favor of the data ("*").

But _long_ before you start having to actually write the data to disk in the swap space, the operating system will start casually forgetting the code. Most of both of these things, while "freshly forgotten" can be "reclaimed" from the disk/page cache.

So when you show me that free memory listing what I see is someone who is bumping against their "minimum desired free memory" limit (e.g. about 1% free ram) and so has a system where it's _possible_ that a good bit of stuff is getting dumped out of the active page tables and into the disk/page cache where it could start bogging down the system in large numbers of short reclaim delays and potentially non-trivial amounts of demand paging.

So not "out of memory" but not ideal. Real measurements of page fault activity and wait for io time needs to be done to determine if more action needs to be taken.

Compare that to my laptop where I've deliberately made sure that memory is always available for fast transient use.

Gust ~ # free
total used free shared buff/cache
available

Mem:        7915804     2820368     2419300       47472     2676136 4790892
Swap:       8778748      311688     8467060

In this configuration when I run bursty stuff I've set aside two gig that sits around "Free" into which the dynamic load can find ample space. (I am not recommending that for you necessarily, were I present I'd do some experimenting). But that dynamic space is where the rsync would be doing its work and so be less likely to stall. (Etc).

Hi,

Capturing "echo t >/proc/magic-sysrq" on waiting for I/O may help you. It shows us where kernel actually be waiting for.

In addition, to confirm whether this problem is caused only by Btrfs or not, the following way can be used.

1. preparing the extra storage,
2. copy Btrfs's data into int by dd if=<LVM volume> of=<extra storage>
3. Use it and confirm whether this problem still happen or not

However, since the size of your Btrfs is quite large, I guess you can't do it. If you have such extra storage, you've already embed it to Btrfs.

Thanks,
Satoru

This actually sounds kind of like the issues I have sometimes on my laptop using btrfs on an SSD, I've mostly resolved them by tuning IO scheduler parameters, as the default IO scheduler (the supposedly Completely Fair Queue, which was obviously named by a mathematician who had never actually run the algorithm) has some pretty brain-dead default settings. The other thing I would suggest looking into regarding the variability is tuning the kernel's write-caching settings, with the defaults you're caching ~1.6G worth of writes before it forces write-back, which is a ridiculous amount; I've that the highest value that is actually usable is about 256M, and that's only if you are doing mostly bursty IO and not the throughput focused stuff that rsync does, I'd say try setting /proc/sys/vm/dirty_background_bytes to 67108864 (64M) and see if that helps things some.

I think you're mistaken. Every diagnostic I've looked at says the opposite.

filesystem (and with a larger xfs one mounted), /proc/meminfo says almost the entirety of the machine's 16GB is free:

  MemTotal:       16469880 kB
  MemFree:        16005392 kB
  MemAvailable:   15974244 kB

Buffers: 84 kB
[...]

Yes, I'm aware of how this works.

Charles

-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

Ok, that's something I will examine. I knew CFQ is completely wrong for SSD use, but I thought it was still one of the better schedulers for spinning disks. Apparently that may not be the case.

Thanks,

Charles

-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

I've already copied the ~16TB of data from the btrfs filesystem to an XFS filesystem. I do not see the performance variability under xfs that I see under btrfs.

Actually, I decided to move to xfs, at least for now. Apparently not many people are using btrfs with filesystems >15TB, so it seems I'm in more-or-less uncharted territory, at least according to the responses I've gotten when looking into this issue.

Charles

-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------