Skip to contentSkip to conversation

btrfs-transacti hangs system for several seconds every few minutes

DISCUSSION Brad Templeton 4 participants 6 messages Mar 28, 2020 – Mar 31, 2020 1 message not in the archive Btrfs file system

lore
Share & tools
lore
#0 lore raw

I have a decent sized 3 disk Raid 1 that I have had on btrfs for many years. Over time, a serious problem has emerged, in that from time to time all I/O will pause, freezing any programs attempting to use the
btrfs filesystem. Performance has degraded over the years as well, so
that just browsing around in directories with 300 or so files often takes many seconds just to autocomplete a filename or do an ls.

But the big problem is that during periods of active but not heavy use, every few minutes the i/o system will hang for periods of 1 to 10
seconds. During these hangs, btrfs-transacti is doing very heavy I/O.
Programs waiting on I/O block -- the most frustrating is typing in vi
and having the echo stop. It's getting close to unusable and may be time to leave btrfs after many years for a different FS.

During these incidents iotop will look like this:

Total DISK READ :     499.57 K/s | Total DISK WRITE :    1639.00 K/s
Actual DISK READ:     492.73 K/s | Actual DISK WRITE:       0.00 B/s

TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
882 be/4 root 499.57 K/s 1604.78 K/s 0.00 % 98.60 %
[btrfs-transacti]
21829 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.23 %
[kworker/u32:1-btrfs-endio-meta]
14662 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.17 %
[kworker/u32:0-btrfs-endio-meta]
22184 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.11 %
[kworker/u32:3-events_freezable_power_]
13063 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.06 %
[kworker/u32:6-events_freezable_power_]

  486 be/3 root        0.00 B/s    6.84 K/s  0.00 %  0.00 % systemd-journald
22213 be/4 brad        0.00 B/s    6.84 K/s  0.00 %  0.00 % chrome

--no-startup-window [ThreadPoolForeg]

A way to reliably generate it, I have found, is to quickly skim through my large video collection (looking for videos) I would be hitting "next" every second or so -- lots of read, but very little write. After doing about 40 seconds of this, it is sure to hang.

I am running kernel 5.3.0 on Ubuntu 18.04.4, but have seen this problem gong back into much older kernels.

My array looks like this:

/dev/sda, ID: 2
Device size: 3.64TiB
Device slack: 0.00B

   Data,RAID1:              1.79TiB
   Metadata,RAID1:          8.00GiB
   Unallocated:             1.84TiB

/dev/sdg, ID: 1
Device size: 9.10TiB
Device slack: 0.00B
Data,RAID1: 7.21TiB
Metadata,RAID1: 14.00GiB
System,RAID1: 32.00MiB
Unallocated: 1.87TiB

/dev/sdh, ID: 3
Device size: 7.28TiB
Device slack: 344.00KiB
Data,RAID1: 5.43TiB
Metadata,RAID1: 8.00GiB
System,RAID1: 32.00MiB
Unallocated: 1.84TiB

/dev/sdg on /home type btrfs (rw,relatime,space_cache,subvolid=256,subvol=/home)

I have 16gb of ram with 16gb of swap on a flash drive, the swap is in use

KiB Mem : 16393944 total, 398800 free, 13538088 used, 2457056 buff/cache
KiB Swap: 16777212 total, 6804352 free, 9972860 used. 2045812 avail Mem

What other information would be useful in attempting to diagnose or fix
this? I like a number of things about BTFS. One of them that I don't
want to give up is the ability to do RAID with different sized disks, which seems like the only way it should work. Switching to ZFS or mdadm again would involve disk upgrades and a very large amount of time copying this much data, but I'll have to do it if I can't diagnose this.

#1 lore raw
Quoted 33 lines · Brad Templeton

I have a decent sized 3 disk Raid 1 that I have had on btrfs for many years. Over time, a serious problem has emerged, in that from time to time all I/O will pause, freezing any programs attempting to use the
btrfs filesystem. Performance has degraded over the years as well, so
that just browsing around in directories with 300 or so files often takes many seconds just to autocomplete a filename or do an ls.

But the big problem is that during periods of active but not heavy use, every few minutes the i/o system will hang for periods of 1 to 10
seconds. During these hangs, btrfs-transacti is doing very heavy I/O.
Programs waiting on I/O block -- the most frustrating is typing in vi
and having the echo stop. It's getting close to unusable and may be time to leave btrfs after many years for a different FS.

During these incidents iotop will look like this:

Total DISK READ :     499.57 K/s | Total DISK WRITE :    1639.00 K/s
Actual DISK READ:     492.73 K/s | Actual DISK WRITE:       0.00 B/s

TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
882 be/4 root 499.57 K/s 1604.78 K/s 0.00 % 98.60 %
[btrfs-transacti]
21829 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.23 %
[kworker/u32:1-btrfs-endio-meta]
14662 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.17 %
[kworker/u32:0-btrfs-endio-meta]
22184 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.11 %
[kworker/u32:3-events_freezable_power_]
13063 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.06 %
[kworker/u32:6-events_freezable_power_]

  486 be/3 root        0.00 B/s    6.84 K/s  0.00 %  0.00 % systemd-journald
22213 be/4 brad        0.00 B/s    6.84 K/s  0.00 %  0.00 % chrome

--no-startup-window [ThreadPoolForeg]

Quoted 7 lines

A way to reliably generate it, I have found, is to quickly skim through my large video collection (looking for videos) I would be hitting "next" every second or so -- lots of read, but very little write. After doing about 40 seconds of this, it is sure to hang.

I am running kernel 5.3.0 on Ubuntu 18.04.4, but have seen this problem gong back into much older kernels.

PSA: Get off 5.3.0. There is a serious bug in kernels 5.1 to 5.4.13 that can lead to metadata corruption resulting in loss of the filesystem. Go to 5.4.14 or later, or back to 4.19.y for y > 100 or so. This advice applies to all btrfs users, it's not related to latency.

In this case 4.19 might be a better choice than later kernels for latency. 5.0 had some latency-related regressions, and fixes for those are still in development.

Quoted 25 lines

My array looks like this:

/dev/sda, ID: 2
Device size: 3.64TiB
Device slack: 0.00B

   Data,RAID1:              1.79TiB
   Metadata,RAID1:          8.00GiB
   Unallocated:             1.84TiB

/dev/sdg, ID: 1
Device size: 9.10TiB
Device slack: 0.00B
Data,RAID1: 7.21TiB
Metadata,RAID1: 14.00GiB
System,RAID1: 32.00MiB
Unallocated: 1.87TiB

/dev/sdh, ID: 3
Device size: 7.28TiB
Device slack: 344.00KiB
Data,RAID1: 5.43TiB
Metadata,RAID1: 8.00GiB
System,RAID1: 32.00MiB
Unallocated: 1.84TiB

Quoted 2 lines

/dev/sdg on /home type btrfs (rw,relatime,space_cache,subvolid=256,subvol=/home)

Two things in the mount options:

1. PSA: Upgrade to space_cache=v2. Unmount the filesystem, then mount it with '-o clear_cache,space_cache=v2' (remount is not sufficient, you have to completely umount). This will take some minutes, but it only has to be done once. Transactions will be quite slow on a filesystem with ~10000 block groups with space_cache=v1. Afterwards, use

btrfs ins dump-tree -t 10 /dev/vgwaya/root |
grep 'owner FREE_SPACE_TREE' | wc -l

to verify the space_cache=v2 conversion was done (it should give a non-zero number). Although directly relevant to this case, this advice is a PSA because it also applies to all btrfs users.

2. Use noatime instead of relatime.

In the mount man page for 'relatime':

since Linux 2.6.30, the file's last access time is always updated if it is more than 1 day old

If you get this high-latency behavior about once a day, but it's fine at other times, then this is the likely cause. Some users need atime updates, and they're usually OK on small SSD filesystems; however, this filesystem is neither small nor SSD, and most users don't need atime.

You didn't mention snapshots. If you don't have snapshots then disregard the rest of this paragraph. If you do have snapshots, then each time you modify a snapshotted subvol (either origin or snapshot, doesn't matter, what matters is that the metadata is shared), btrfs will be doing extra writes to unshare shared pages and update reference counts. Immediately after the snapshot is created, the write multiplication factor is about 300. The factor drops rapidly to 1.0, but it can take a few minutes to get through the first 10000 page updates after a snapshot, and you can easily get that many by touching 500 files. Note that the snapshot could have been made in the past, its existence will still affect the write performance of the filesystem in the present.

All of the above effects combine: 5.0 and later do not attempt to manage latency, atime updates throw a lot of writes into the queue at once, space_cache=v1 makes every write slower to exit the queue, and fresh snapshots multiply everything else by an order of magnitude. With all of those at once, I'm surprised it's as fast as you reported. Starting with kernel 5.0 it's not hard to make a btrfs commit take 10 hours.

Quoted 4 lines

I have 16gb of ram with 16gb of swap on a flash drive, the swap is in use

KiB Mem : 16393944 total, 398800 free, 13538088 used, 2457056 buff/cache
KiB Swap: 16777212 total, 6804352 free, 9972860 used. 2045812 avail Mem

Check slabtop:

	# slabtop | grep btrfs_delayed_ref_head
	105072 105072 100%    0.33K   8756       12     35024K btrfs_delayed_ref_head

Divide the second number (count of btrfs_delayed_ref_head slabs in use) by about 1000 (depends on how fast your disks are, range is about 500 to 10000 for consumer hardware) and the result is roughly the commit latency in seconds. It's not the only time spent in a commit, but btrfs spends orders of magnitude more time on delayed refs than on anything else. On kernels before 5.0 btrfs kept the delayed ref head count below 10000, but after 5.0 it is allowed to grow until memory is exhausted.

The latency fixes currently in development put the latency caps from 4.19 back in, and also add new ones, e.g. snapshot delete could create unlimited latency in btrfs since the beginning. 5.7 or 5.8 should be better at latency than 4.19.

Quoted 6 lines

What other information would be useful in attempting to diagnose or fix
this? I like a number of things about BTFS. One of them that I don't
want to give up is the ability to do RAID with different sized disks, which seems like the only way it should work. Switching to ZFS or mdadm again would involve disk upgrades and a very large amount of time copying this much data, but I'll have to do it if I can't diagnose this.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 7 lines · Brad Templeton

Not using qgroups.  Not doing snapshots.    Did a reboot with the options to upgrade to v2 -- it failed, in that the disk check took more than 6 minutes, but it worked, and the second time I was able to boot, and -- knock on wood -- so far it has not hung.

I wonder why they put 5.3.0 as the standard advanced Kernel in Ubuntu LTS if it has a data corruption bug.

Ubuntu, like most Linux distros, chooses the kernel version to ship based on release date. QA/defect data does not usually affect that decision.

Starting with 5.2 there are internal btrfs mitigations for metadata corruption. 5.3, 5.4, and 5.5 contain improved mitigations which are able to detect more corruption cases introduced by regressions in 5.1 (and also external causes like RAM failure), and block the corrupted data before it reaches the disk. This blocking is done by forcing the filesystem readonly, so even if you don't get on-disk corruption, you're still going to need to at least unmount the filesystem and possibly reboot if you hit the bug.

The risk is that the tree checker can't detect every possible corruption, and some things might slip through undetected. The probability of this failure is normally small (about the same failure rate as an average hard drive), but the risk increases significantly while metadata-intensive operations like device delete or shrinking resize are running. On a 10TB filesystem it is unlikely you would be able to complete a device delete operation without some kind of error, and then you're rolling dice to see if the filesystem survives intact.

Ubuntu could backport the fix from 5.4.14 (the patch applies to 5.1 and later). I don't know if Ubuntu has done this. When in doubt, assume the fix is not present.

Quoted 4 lines

I don't know if I've seen any release
of 5.4.14 in a PPA yet -- manual kernel install is such a pain the few times I have done it.  I could revert, but the reason I switched to 5.3, not long ago, was another problem with sound drivers.

Yeah, non-overlapping bug lifetimes suck.

Quoted 3 lines

BTW, even though it now works, it still takes 90 seconds every boot doing a disk check, even after what I think is a clean shutdown.   I presume that is not normal, any clues on what may cause that?

It shouldn't be doing a disk check at all, even on an unclean shutdown. btrfs writes data in order so no checking is required unless your disks break (and with raid1 you need two broken disks at the same time). fsck.btrfs is a no-op stub.

90 seconds sounds about right for the block group scan when mounting on a 10TB filesystem. There's a feature called block group tree in kernel 5.5 that helps with that: it lays out block group items on disk closer together so they can be read in milliseconds. This is an on-disk format change, so once you enable that feature, you wouldn't be able to mount the filesystem on an older kernel. This can be a problem if your sound drivers have regressions. You might want to wait a few kernel releases to be sure you don't need to downgrade.

Quoted 160 lines · Zygo Blaxell

On Sat, Mar 28, 2020 at 11:26:56AM -0700, Brad Templeton wrote:

I have a decent sized 3 disk Raid 1 that I have had on btrfs for many years. Over time, a serious problem has emerged, in that from time to time all I/O will pause, freezing any programs attempting to use the
btrfs filesystem. Performance has degraded over the years as well, so
that just browsing around in directories with 300 or so files often takes many seconds just to autocomplete a filename or do an ls.

But the big problem is that during periods of active but not heavy use, every few minutes the i/o system will hang for periods of 1 to 10
seconds. During these hangs, btrfs-transacti is doing very heavy I/O.
Programs waiting on I/O block -- the most frustrating is typing in vi
and having the echo stop. It's getting close to unusable and may be time to leave btrfs after many years for a different FS.

During these incidents iotop will look like this:

 Total DISK READ :     499.57 K/s | Total DISK WRITE :    1639.00 K/s
 Actual DISK READ:     492.73 K/s | Actual DISK WRITE:       0.00 B/s
   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
   882 be/4 root      499.57 K/s 1604.78 K/s  0.00 % 98.60 %
 [btrfs-transacti]
 21829 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.23 %
 [kworker/u32:1-btrfs-endio-meta]
 14662 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.17 %
 [kworker/u32:0-btrfs-endio-meta]
 22184 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.11 %
 [kworker/u32:3-events_freezable_power_]
 13063 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.06 %
 [kworker/u32:6-events_freezable_power_]
   486 be/3 root        0.00 B/s    6.84 K/s  0.00 %  0.00 % systemd-journald
 22213 be/4 brad        0.00 B/s    6.84 K/s  0.00 %  0.00 % chrome

--no-startup-window [ThreadPoolForeg]

A way to reliably generate it, I have found, is to quickly skim through my large video collection (looking for videos) I would be hitting "next" every second or so -- lots of read, but very little write. After doing about 40 seconds of this, it is sure to hang.

I am running kernel 5.3.0 on Ubuntu 18.04.4, but have seen this problem gong back into much older kernels.

PSA: Get off 5.3.0. There is a serious bug in kernels 5.1 to 5.4.13 that can lead to metadata corruption resulting in loss of the filesystem. Go to 5.4.14 or later, or back to 4.19.y for y > 100 or so. This advice applies to all btrfs users, it's not related to latency.

In this case 4.19 might be a better choice than later kernels for latency. 5.0 had some latency-related regressions, and fixes for those are still in development.

My array looks like this:

/dev/sda, ID: 2
Device size: 3.64TiB
Device slack: 0.00B
Data,RAID1: 1.79TiB
Metadata,RAID1: 8.00GiB
Unallocated: 1.84TiB

/dev/sdg, ID: 1
Device size: 9.10TiB
Device slack: 0.00B
Data,RAID1: 7.21TiB
Metadata,RAID1: 14.00GiB
System,RAID1: 32.00MiB
Unallocated: 1.87TiB

/dev/sdh, ID: 3
Device size: 7.28TiB
Device slack: 344.00KiB
Data,RAID1: 5.43TiB
Metadata,RAID1: 8.00GiB
System,RAID1: 32.00MiB
Unallocated: 1.84TiB

/dev/sdg on /home type btrfs (rw,relatime,space_cache,subvolid=256,subvol=/home)

Two things in the mount options:

1. PSA: Upgrade to space_cache=v2. Unmount the filesystem, then mount it with '-o clear_cache,space_cache=v2' (remount is not sufficient, you have to completely umount). This will take some minutes, but it only has to be done once. Transactions will be quite slow on a filesystem with ~10000 block groups with space_cache=v1. Afterwards, use

btrfs ins dump-tree -t 10 /dev/vgwaya/root |
grep 'owner FREE_SPACE_TREE' | wc -l

to verify the space_cache=v2 conversion was done (it should give a non-zero number). Although directly relevant to this case, this advice is a PSA because it also applies to all btrfs users.

2. Use noatime instead of relatime.

In the mount man page for 'relatime':

since Linux 2.6.30, the file's last access time is always updated if it is more than 1 day old

If you get this high-latency behavior about once a day, but it's fine at other times, then this is the likely cause. Some users need atime updates, and they're usually OK on small SSD filesystems; however, this filesystem is neither small nor SSD, and most users don't need atime.

You didn't mention snapshots. If you don't have snapshots then disregard the rest of this paragraph. If you do have snapshots, then each time you modify a snapshotted subvol (either origin or snapshot, doesn't matter, what matters is that the metadata is shared), btrfs will be doing extra writes to unshare shared pages and update reference counts. Immediately after the snapshot is created, the write multiplication factor is about 300. The factor drops rapidly to 1.0, but it can take a few minutes to get through the first 10000 page updates after a snapshot, and you can easily get that many by touching 500 files. Note that the snapshot could have been made in the past, its existence will still affect the write performance of the filesystem in the present.

All of the above effects combine: 5.0 and later do not attempt to manage latency, atime updates throw a lot of writes into the queue at once, space_cache=v1 makes every write slower to exit the queue, and fresh snapshots multiply everything else by an order of magnitude. With all of those at once, I'm surprised it's as fast as you reported. Starting with kernel 5.0 it's not hard to make a btrfs commit take 10 hours.

I have 16gb of ram with 16gb of swap on a flash drive, the swap is in use

KiB Mem : 16393944 total, 398800 free, 13538088 used, 2457056 buff/cache
KiB Swap: 16777212 total, 6804352 free, 9972860 used. 2045812 avail Mem

Check slabtop:

         # slabtop | grep btrfs_delayed_ref_head
         105072 105072 100%    0.33K   8756       12     35024K btrfs_delayed_ref_head

Divide the second number (count of btrfs_delayed_ref_head slabs in use) by about 1000 (depends on how fast your disks are, range is about 500 to 10000 for consumer hardware) and the result is roughly the commit latency in seconds. It's not the only time spent in a commit, but btrfs spends orders of magnitude more time on delayed refs than on anything else. On kernels before 5.0 btrfs kept the delayed ref head count below 10000, but after 5.0 it is allowed to grow until memory is exhausted.

The latency fixes currently in development put the latency caps from 4.19 back in, and also add new ones, e.g. snapshot delete could create unlimited latency in btrfs since the beginning. 5.7 or 5.8 should be better at latency than 4.19.

What other information would be useful in attempting to diagnose or fix
this? I like a number of things about BTFS. One of them that I don't
want to give up is the ability to do RAID with different sized disks, which seems like the only way it should work. Switching to ZFS or mdadm again would involve disk upgrades and a very large amount of time copying this much data, but I'll have to do it if I can't diagnose this.

#4 lore raw
Quoted 9 lines · Zygo Blaxell

90 seconds sounds about right for the block group scan when mounting on a 10TB filesystem. There's a feature called block group tree in kernel 5.5 that helps with that: it lays out block group items on disk closer together so they can be read in milliseconds. This is an on-disk format change, so once you enable that feature, you wouldn't be able to mount the filesystem on an older kernel. This can be a problem if your sound drivers have regressions. You might want to wait a few kernel releases to be sure you don't need to downgrade.

I'm not seeing anything about block group tree in btrfs/super.c.

There is block_group_cache_tree but I'm not seeing anything about it in 'man 5 btrfs' using btrfs-progs 5.4, or in the devel branch.

So I'm not sure what mount option or btrfstune option this would be, seems to be automatic?
https://github.com/kdave/btrfs-progs/commit/2eaf862f46b3ccb6b7248a0417ebf7096bc93b80

--
Chris Murphy

#5 lore raw
Quoted 12 lines · Chris Murphy
Quoted 9 lines · Zygo Blaxell

90 seconds sounds about right for the block group scan when mounting on a 10TB filesystem. There's a feature called block group tree in kernel 5.5 that helps with that: it lays out block group items on disk closer together so they can be read in milliseconds. This is an on-disk format change, so once you enable that feature, you wouldn't be able to mount the filesystem on an older kernel. This can be a problem if your sound drivers have regressions. You might want to wait a few kernel releases to be sure you don't need to downgrade.

Quoted 8 lines

I'm not seeing anything about block group tree in btrfs/super.c.

There is block_group_cache_tree but I'm not seeing anything about it in 'man 5 btrfs' using btrfs-progs 5.4, or in the devel branch.

So I'm not sure what mount option or btrfstune option this would be, seems to be automatic?
https://github.com/kdave/btrfs-progs/commit/2eaf862f46b3ccb6b7248a0417ebf7096bc93b80

Sorry, my mistake...it was in one of the misc-next branches, but seems to have been dropped. Maybe not finished yet?

Quoted 4 lines

--
Chris Murphy

#6 lore raw
In reply to Brad Templeton
Quoted 7 lines · Brad Templeton

I have a decent sized 3 disk Raid 1 that I have had on btrfs for many years. Over time, a serious problem has emerged, in that from time to time all I/O will pause, freezing any programs attempting to use the
btrfs filesystem. Performance has degraded over the years as well, so
that just browsing around in directories with 300 or so files often takes many seconds just to autocomplete a filename or do an ls.

Quoted 6 lines

But the big problem is that during periods of active but not heavy use, every few minutes the i/o system will hang for periods of 1 to 10
seconds. During these hangs, btrfs-transacti is doing very heavy I/O.
Programs waiting on I/O block -- the most frustrating is typing in vi
and having the echo stop. It's getting close to unusable and may be time to leave btrfs after many years for a different FS.

Are you using qgroups and doing routinely balance or snapshot drop?

Qgroup is known for causing a lot of performance impact, especially for snapshot drop and balance.
For the balance part, it get improved in recent releases, but v5.3 it shouldn't cause too much overhead unless it's doing a lot of background IO during balance.

Anyway, if you're using qgroup and it's not critical to your use case, disabling qgroup would help a lot.

Thanks,
Qu

Quoted 70 lines

During these incidents iotop will look like this:

Total DISK READ :     499.57 K/s | Total DISK WRITE :    1639.00 K/s
Actual DISK READ:     492.73 K/s | Actual DISK WRITE:       0.00 B/s

TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
882 be/4 root 499.57 K/s 1604.78 K/s 0.00 % 98.60 %
[btrfs-transacti]
21829 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.23 %
[kworker/u32:1-btrfs-endio-meta]
14662 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.17 %
[kworker/u32:0-btrfs-endio-meta]
22184 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.11 %
[kworker/u32:3-events_freezable_power_]
13063 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.06 %
[kworker/u32:6-events_freezable_power_]

  486 be/3 root        0.00 B/s    6.84 K/s  0.00 %  0.00 % systemd-journald
22213 be/4 brad        0.00 B/s    6.84 K/s  0.00 %  0.00 % chrome

--no-startup-window [ThreadPoolForeg]

A way to reliably generate it, I have found, is to quickly skim through my large video collection (looking for videos) I would be hitting "next" every second or so -- lots of read, but very little write. After doing about 40 seconds of this, it is sure to hang.

I am running kernel 5.3.0 on Ubuntu 18.04.4, but have seen this problem gong back into much older kernels.

My array looks like this:

/dev/sda, ID: 2
Device size: 3.64TiB
Device slack: 0.00B

   Data,RAID1:              1.79TiB
   Metadata,RAID1:          8.00GiB
   Unallocated:             1.84TiB

/dev/sdg, ID: 1
Device size: 9.10TiB
Device slack: 0.00B
Data,RAID1: 7.21TiB
Metadata,RAID1: 14.00GiB
System,RAID1: 32.00MiB
Unallocated: 1.87TiB

/dev/sdh, ID: 3
Device size: 7.28TiB
Device slack: 344.00KiB
Data,RAID1: 5.43TiB
Metadata,RAID1: 8.00GiB
System,RAID1: 32.00MiB
Unallocated: 1.84TiB

/dev/sdg on /home type btrfs (rw,relatime,space_cache,subvolid=256,subvol=/home)

I have 16gb of ram with 16gb of swap on a flash drive, the swap is in use

KiB Mem : 16393944 total, 398800 free, 13538088 used, 2457056 buff/cache
KiB Swap: 16777212 total, 6804352 free, 9972860 used. 2045812 avail Mem

What other information would be useful in attempting to diagnose or fix
this? I like a number of things about BTFS. One of them that I don't
want to give up is the ability to do RAID with different sized disks, which seems like the only way it should work. Switching to ZFS or mdadm again would involve disk upgrades and a very large amount of time copying this much data, but I'll have to do it if I can't diagnose this.