btrfs scrub ioprio

TL;DR scrub's ioprio argument isn't really helpful - a scrub murders system performance til it's done.

My system:

3.11 kernel (from Ubuntu Saucy)
btrfs-tools from 2013-07 (from Debian Sid)
Opteron 8-core CPU
32GB RAM
4 WD 1TB Black drives in a btrfs RAID10 (data and metadata).

iotop shows that the idle priority really is set on the processes:

But unfortunately, despite being on "idle" priority, the scrub is KILLING system performance. This is one of my eight CPU cores. Any core with a significant amount of non-idle time has three to four times as much wait time as it does user/system time. Brutal:

Anybody got any ideas that might make the scrub go a little less... enthusiastically... and leave the system more usable? Right now, with a scrub running, the system is so burdened that it may take as much as two to three seconds to so much as display a manpage. God help you if you want to, say, log into a VM running on the system. It'll GET there, but it'll look like it's iSCSI on TCP over carrier pigeon.

A large part of this might be that the idle I/O scheduling class does not put a limit on bandwidth utilization, so even though it is using 'idle' bandwidth, it is actually using all of the bandwidth that is 'unused' by other tasks, which kills your disk latency. As i see it, the best option is probably either:
1. set up something to suspend the scrub if there are other pending I/O requests (which would probably need kernel support)
2. allow the user to set a reasonable I/O bandwidth limit on the scrub processes (you could already do this with the BIO cgroup, but it would be nice to not need that to be compiled into the kernel to have this happen)

Can you elaborate on this please? I'm not directly familiar with cgroups, I'd greatly appreciate a quick-and-dirty example of using BIO cgroup to limit I/O bandwidth.

Limiting bandwidth definitely would ameliorate the problem for me; I already use pv's bw-limiting feature to make btrfs send | btrfs receive tolerable.

I'm not very clear on the specifics, (the only cgroup I personally use with any regularity is freezer), but I would suggest looking at Documentation/cgroups/blkio-controller.txt in the kernel source tree (cgroups are one of the better subsystems WRT documentation). I'm pretty certain that most distributions have the required kernel options compiled in by default, and automatically mount the needed filesystems.

I don't run Ubuntu, but *maybe* they use the deadline IO scheduler by default, which does not handle IO priorities (by design). So maybe make sure your devices don't say "deadline" in /sys/block/sdX/queue/scheduler, and if they do, set them to cfq.

-h

OTOH I've switched from CFQ to Deadline in Gentoo and IO responsiveness has greatly improved since then (spinning rust). The system boots faster, is able to start programs faster during IO stress, but swapping seems to stress the system a lot more. But since I got 16 GB RAM this is not a big problem. Throughput is not as good as with CFQ - but this is a desktop system:
latency counts more than throughput, and that improved a lot.

I've tried BFQ before switching to Deadline but BFQ just does not cut it. During high IO loads it simply does not feel interactive although it was designed to guarantee exactly that. The fact that it also made my kernel unstable (with bfq related backtraces in the kernel log) I've switched away from it.

But I have to admit that I'm using the following setting to reduce processes blocking on high IO loads, tho you should have enough RAM to use it:

$ cat /etc/local.d/nr_requests.start 
#!/bin/sh

for nr_requests in /sys/block/sd[abc]/queue/nr_requests; do

        echo -n 4096 >$nr_requests

done

My btrfs pool spans sd[abc].

I'm using a weekly scrubbing job and a daily rsync backup job and see no drawbacks from using deadline scheduler, rather it improved a lot.

@OP:

I suggest looking at the surrounding parameters also for trouble shooting:
Maybe the SATA driver just doesn't cut it? Maybe a BIOS setting is wrong (AHCI mode?)... Does your system run low on RAM (with deadline this can become a heavy bottleneck)? Also, 3.12 kernel improved btrfs performance, you may want to try that first.

Did you check which IO scheduler is in use?

BTW: Yes, Ubuntu has switched from CFQ to Deadline by default for servers and desktops - with the same finding I had when trying. CFQ is more or less optimized for the general usecase and tries to behave like the old anticipatory scheduler, but it is designed with "fair" bandwidth distribution in mind which hurts interactivity a lot. But desktop systems feel fast by reducing latency, not by improving or fair-sharing bandwidth. Most servers need low latency for good service response times, throughput is not important. So deadline is not a bad choice for most use-cases (I know, "most use-cases" is subjective but I think for the edge-cases which benefit from CFQ the user already knows how to switch to the right scheduler). We've switched all our virtualized servers from CFQ to noop and they are running a lot better now with much decreased IO latency and lower IO wait. I think "fair bandwidth distribution" is just no longer that important with devices being able to deliver 100+ MB/s of bandwidth and low latency becomes more and more important...

But: YMMV... ;-)

Regards,
Kai