Trying to balance/delete after drive failure

Hi Guys,

I am running a raid1 btrfs with 4 disks. One of my disks died the other day. So I replaced it with a new one. After that I tried to delete the failed (now missing) disk. This resulted in some but not much IO and some messages like these:

kernel: BTRFS info (device sdd): found 9 extents
kernel: [22138.202334] BTRFS info (device sdd): relocating block group 10093403832320 flags 17

Perf show this:

  47.21%  [kernel]                    [k] rb_next
  29.89%  [kernel]                    [k] comp_entry
   9.54%  [kernel]                    [k] btrfs_merge_delayed_refs

And top this:
27742 root 20 0 0 0 0 R 100.0 0.0 19:22.90 kworker/u8:7

But at one point no new messages appeared and no IO could be seen. I thought maybe the process hung. So I rebooted the machine. After that I tried a balance with the same result.

Here some information that might be important:

uname -a
Linux thales 4.1.5-100.fc21.x86_64 #1 SMP Tue Aug 11 00:24:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

###

btrfs --version
btrfs-progs v4.1

###

btrfs --version
btrfs-progs v4.1
[root@thales ~]# btrfs fi show
Label: 'data' uuid: 1f7e44b4-39ed-475d-b341-93b2e44f8f89
Total devices 5 FS bytes used 3.40TiB
devid 1 size 2.73TiB used 2.58TiB path /dev/sdb
devid 4 size 931.51GiB used 779.00GiB path /dev/sde
devid 5 size 931.51GiB used 779.00GiB path /dev/sdd
devid 6 size 5.46TiB used 379.00GiB path /dev/sda
*** Some devices missing

btrfs-progs v4.1

###

btrfs fi df /mnt/data/
Data, RAID1: total=3.46TiB, used=3.40TiB System, RAID1: total=32.00MiB, used=528.00KiB Metadata, RAID1: total=9.00GiB, used=7.63GiB GlobalReserve, single: total=512.00MiB, used=15.74MiB

###

dmesg
[...]

[33209.949992] BTRFS info (device sdd): found 11 extents
[33244.802181] BTRFS info (device sdd): found 11 extents
[33247.178590] BTRFS info (device sdd): relocating block group 9826042118144 flags 20

###

Thanks for your help,
Jan

jan posted on Mon, 24 Aug 2015 11:20:41 +0000 as excerpted:

FWIW, one-for-one replace is what the (relatively new) btrfs replace (note, NOT btrfs device replace, as might have been expected) command is for, as it shortcuts the add/delete steps a bit. See the btrfs-replace manpage.

I had delayed replying to this in the hope that someone else who could perhaps be of more help would reply, but since I don't see any replies yet...

First, let me congratulate you on being actually mostly current. You're slightly behind on btrfs-progs, with 4.1.2 being the latest, but nothing major there, and are on the latest stable 4.1 kernel series, so good going! =:^)

How many btrfs snapshots, if any, on the filesystem, and do you have btrfs quotas enabled on it at all?

I ask because btrfs maintenance such as balance, either directly invoked, or invoked with a btrfs device delete or btrfs replace, doesn't scale particularly well in the presence of large numbers of snapshots or with quotas enabled. In these instances, the balance will appear to have stopped in terms of IO, but it's still doing heavy CPU work figuring out the mapping between all those snapshots (and/or quotagroups) and the extents it happens to be working on at the time.

Generally, I recommend no more than 10K snapshots at absolute most, and if at all possible, keep it to 1-2K snapshots. Even with say half-hourly snapshots, given reasonable thinning, 250-ish snapshots per subvolume is quite reasonable, thus allowing four subvolumes at the full 250 snapshots schedule for the 1000 snapshot per filesystem cap, or eight subvolumes worth for the 2K cap. And quotas are still problematic and unreliable on btrfs for other reasons as well, tho they're working on it, so there I recommend that people use other more mature filesystems if they really need quotas, or keep them off on btrfs, if not, unless of course they're working with the devs to test the quota feature specifically.

Between that and the several TiB of raw data and metadata shown in your btrfs fi show and df, it could take awhile, tho if you've only a handful of snapshots and don't have quotas enabled, IO shouldn't be stalled for /too/ long and it shouldn't take /forever/.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman