Post ext3 conversion problems

Hi, all. I've been playing around with an old laptop of mine, and I figured I'd use it as a learning / bugfinding opportunity. Its /home partition was originally ext3. I have a full partition image of this drive as a backup, so I can do (and have done) potentially destructive things. The system disk is a ~6 year old SSD.

To start, I rebooted to a livedisk (Arch, kernel 4.7.2 w/progs 4.7.1) and ran a simple btrfs-convert on it. After patching up the fstab and rebooting, everything seemed fine. I deleted the recovery subvol, ran a full balance, ran a full defrag, and rebooted again. I then decided to try (as an experiment) using DUP mode for data and metadata. I ran that balance without issue, then started using the machine. Sometime later, I got the following remount ro:

[ 7316.764235] ------------[ cut here ]------------
[ 7316.764292] WARNING: CPU: 2 PID: 14196 at fs/btrfs/inode.c:2954 btrfs_finish_ordered_io+0x6bc/0x6d0 [btrfs]
[ 7316.764297] BTRFS: Transaction aborted (error -95)
[ 7316.764301] Modules linked in: fuse sha256_ssse3 sha256_generic hmac drbg ansi_cprng ctr ccm joydev mousedev uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media crc32c_generic iTCO_wdt btrfs iTCO_vendor_support arc4 xor ath9k raid6_pq ath9k_common ath9k_hw ath mac80211 snd_hda_codec_realtek snd_hda_codec_generic psmouse input_leds coretemp snd_hda_intel led_class pcspkr snd_hda_codec cfg80211 snd_hwdep snd_hda_core snd_pcm lpc_ich snd_timer atl1c rfkill snd soundcore shpchp intel_agp wmi thermal fjes battery evdev ac tpm_tis mac_hid tpm sch_fq_codel vboxnetflt(O) vboxnetadp(O) pci_stub vboxpci(O) vboxdrv(O) loop sg acpi_cpufreq ip_tables x_tables ext4 crc16 jbd2 mbcache sd_mod serio_raw atkbd libps2 ahci libahci uhci_hcd libata scsi_mod ehci_pci ehci_hcd usbcore
[ 7316.764434]  usb_common i8042 serio i915 video button intel_gtt i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
[ 7316.764462] CPU: 2 PID: 14196 Comm: kworker/u8:11 Tainted: G           O    4.7.3-5-ck #1
[ 7316.764467] Hardware name: ASUSTeK Computer INC. 1015PEM/1015PE, BIOS 0903    11/08/2010
[ 7316.764507] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
[ 7316.764513]  0000000000000286 000000006101f47d ffff8800230dbc78 ffffffff812f0215
[ 7316.764522]  ffff8800230dbcc8 0000000000000000 ffff8800230dbcb8 ffffffff8107ae6f
[ 7316.764530]  00000b8a00000035 ffff88007791afa8 ffff8800751d9000 ffff880014101d40
[ 7316.764538] Call Trace:
[ 7316.764551]  [<ffffffff812f0215>] dump_stack+0x63/0x8e
[ 7316.764560]  [<ffffffff8107ae6f>] __warn+0xcf/0xf0
[ 7316.764567]  [<ffffffff8107aef1>] warn_slowpath_fmt+0x61/0x80
[ 7316.764605]  [<ffffffffa07aa362>] ? unpin_extent_cache+0xa2/0xf0 [btrfs]
[ 7316.764640]  [<ffffffffa07628e6>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 7316.764677]  [<ffffffffa079aaac>] btrfs_finish_ordered_io+0x6bc/0x6d0 [btrfs]
[ 7316.764715]  [<ffffffffa079adc5>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 7316.764753]  [<ffffffffa07c5f8e>] btrfs_scrubparity_helper+0x7e/0x360 [btrfs]
[ 7316.764791]  [<ffffffffa07c62fe>] btrfs_endio_write_helper+0xe/0x10 [btrfs]
[ 7316.764799]  [<ffffffff810949bd>] process_one_work+0x1ed/0x490
[ 7316.764806]  [<ffffffff81094ca9>] worker_thread+0x49/0x500
[ 7316.764813]  [<ffffffff81094c60>] ? process_one_work+0x490/0x490
[ 7316.764820]  [<ffffffff8109ac3a>] kthread+0xda/0xf0
[ 7316.764830]  [<ffffffff815c553f>] ret_from_fork+0x1f/0x40
[ 7316.764838]  [<ffffffff8109ab60>] ? kthread_worker_fn+0x170/0x170
[ 7316.764843] ---[ end trace 90f54effc5e294b0 ]---
[ 7316.764851] BTRFS: error (device sda2) in btrfs_finish_ordered_io:2954: errno=-95 unknown
[ 7316.764859] BTRFS info (device sda2): forced readonly
[ 7316.765396] pending csums is 9437184

After seeing this, I decided to attempt a repair (confident that I could restore from backup if it failed). At the time, I was unaware of the issues with progs 4.7.1, so when I ran the check and saw all the incorrect backrefs messages, I figured that was my problem and ran the --repair. Of course, this didn't make the messages go away on subsequent checks, so I looked further and found this bug:

https://bugzilla.kernel.org/show_bug.cgi?id=155791

I updated progs to 4.7.2 and re-ran the --repair (I didn't save any of the logs from these, unfortunately). The repair seemed to work (I also used --init-extent-tree), as current checks don't report any errors.

The system boots and mounts the FS just fine. I can read from it all day, scrubs complete without failure, but just using the system for a while will eventually trigger the same "Transaction aborted (error -95)" error.

I realize this is something of a mess, and that I was less than methodical with my actions so far. Given that I have a full backup that can be restored if need be (and I certainly could try running the convert again), what is my best course of action?

Not a mess, I think it's a good bug report. I think Qu and David know more about the latest iteration of the convert code. If you can wait until next week at least to see if they have questions that'd be best. If you need to get access to the computer sooner than later I suggest btrfs-image -c9 -t4 -s to make a filename sanitized copy of the filesystem metadata for them to look at, just in case. They might be able to figure out the problem just from the stack trace, but better to have the image before blowing away the file system, just in case they want it.

In the mean time, is there any way to make the kernel more verbose about btrfs errors? It would be nice to see, for example, what was in the transaction that failed, or at least what files / metadata it was touching.

Quoted 4 lines · Sean Greenslade

In the mean time, is there any way to make the kernel more verbose about btrfs errors? It would be nice to see, for example, what was in the transaction that failed, or at least what files / metadata it was touching.

No idea. Maybe one of the compile time options:

CONFIG_BTRFS_FS_CHECK_INTEGRITY=y
This also requires mount options, either check_int or check_int_data CONFIG_BTRFS_FS_RUN_SANITY_TESTS
CONFIG_BTRFS_DEBUG=y
https://patchwork.kernel.org/patch/846462/
CONFIG_BTRFS_ASSERT=y

Actually, even before that maybe if you did a 'btrfs-debug-tree /dev/sdX'

That might explode in the vicinity of the problem. Thing is, btrfs check doesn't see anything wrong with the metadata, so chances are debug-tree won't either.

Quoted 7 lines · Sean Greenslade

Hi, all. I've been playing around with an old laptop of mine, and I figured I'd use it as a learning / bugfinding opportunity. Its /home partition was originally ext3. I have a full partition image of this drive as a backup, so I can do (and have done) potentially destructive things. The system disk is a ~6 year old SSD.

To start, I rebooted to a livedisk (Arch, kernel 4.7.2 w/progs 4.7.1)

Quoted 17 lines

and ran a simple btrfs-convert on it. After patching up the fstab and rebooting, everything seemed fine. I deleted the recovery subvol, ran a full balance, ran a full defrag, and rebooted again. I then decided to try (as an experiment) using DUP mode for data and metadata. I ran that balance without issue, then started using the machine. Sometime later, I got the following remount ro:

[ 7316.764235] ------------[ cut here ]------------
[ 7316.764292] WARNING: CPU: 2 PID: 14196 at fs/btrfs/inode.c:2954 btrfs_finish_ordered_io+0x6bc/0x6d0 [btrfs]
[ 7316.764297] BTRFS: Transaction aborted (error -95)
[ 7316.764301] Modules linked in: fuse sha256_ssse3 sha256_generic hmac drbg ansi_cprng ctr ccm joydev mousedev uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media crc32c_generic iTCO_wdt btrfs iTCO_vendor_support arc4 xor ath9k raid6_pq ath9k_common ath9k_hw ath mac80211 snd_hda_codec_realtek snd_hda_codec_generic psmouse input_leds coretemp snd_hda_intel led_class pcspkr snd_hda_codec cfg80211 snd_hwdep snd_hda_core snd_pcm lpc_ich snd_timer atl1c rfkill snd soundcore shpchp intel_agp wmi thermal fjes battery evdev ac tpm_tis mac_hid tpm sch_fq_codel vboxnetflt(O) vboxnetadp(O) pci_stub vboxpci(O) vboxdrv(O) loop sg acpi_cpufreq ip_tables x_tables ext4 crc16 jbd2 mbcache sd_mod serio_raw atkbd libps2 ahci libahci uhci_hcd libata scsi_mod ehci_pci ehci_hcd usbcore
[ 7316.764434]  usb_common i8042 serio i915 video button intel_gtt i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
[ 7316.764462] CPU: 2 PID: 14196 Comm: kworker/u8:11 Tainted: G           O    4.7.3-5-ck #1
[ 7316.764467] Hardware name: ASUSTeK Computer INC. 1015PEM/1015PE, BIOS 0903    11/08/2010
[ 7316.764507] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
[ 7316.764513]  0000000000000286 000000006101f47d ffff8800230dbc78 ffffffff812f0215
[ 7316.764522]  ffff8800230dbcc8 0000000000000000 ffff8800230dbcb8 ffffffff8107ae6f

Quoted 8 lines

[ 7316.764530]  00000b8a00000035 ffff88007791afa8 ffff8800751d9000 ffff880014101d40
[ 7316.764538] Call Trace:
[ 7316.764551]  [<ffffffff812f0215>] dump_stack+0x63/0x8e
[ 7316.764560]  [<ffffffff8107ae6f>] __warn+0xcf/0xf0
[ 7316.764567]  [<ffffffff8107aef1>] warn_slowpath_fmt+0x61/0x80
[ 7316.764605]  [<ffffffffa07aa362>] ? unpin_extent_cache+0xa2/0xf0 [btrfs]
[ 7316.764640]  [<ffffffffa07628e6>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 7316.764677]  [<ffffffffa079aaac>] btrfs_finish_ordered_io+0x6bc/0x6d0 [btrfs]

Quoted 2 lines

[ 7316.764715]  [<ffffffffa079adc5>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 7316.764753]  [<ffffffffa07c5f8e>] btrfs_scrubparity_helper+0x7e/0x360 [btrfs]

Quoted 9 lines

[ 7316.764791]  [<ffffffffa07c62fe>] btrfs_endio_write_helper+0xe/0x10 [btrfs]
[ 7316.764799]  [<ffffffff810949bd>] process_one_work+0x1ed/0x490
[ 7316.764806]  [<ffffffff81094ca9>] worker_thread+0x49/0x500
[ 7316.764813]  [<ffffffff81094c60>] ? process_one_work+0x490/0x490
[ 7316.764820]  [<ffffffff8109ac3a>] kthread+0xda/0xf0
[ 7316.764830]  [<ffffffff815c553f>] ret_from_fork+0x1f/0x40
[ 7316.764838]  [<ffffffff8109ab60>] ? kthread_worker_fn+0x170/0x170
[ 7316.764843] ---[ end trace 90f54effc5e294b0 ]---
[ 7316.764851] BTRFS: error (device sda2) in btrfs_finish_ordered_io:2954: errno=-95 unknown

Quoted 10 lines

[ 7316.764859] BTRFS info (device sda2): forced readonly
[ 7316.765396] pending csums is 9437184

After seeing this, I decided to attempt a repair (confident that I could restore from backup if it failed). At the time, I was unaware of the issues with progs 4.7.1, so when I ran the check and saw all the incorrect backrefs messages, I figured that was my problem and ran the --repair. Of course, this didn't make the messages go away on subsequent checks, so I looked further and found this bug:

Quoted 5 lines

https://bugzilla.kernel.org/show_bug.cgi?id=155791

I updated progs to 4.7.2 and re-ran the --repair (I didn't save any of the logs from these, unfortunately). The repair seemed to work (I also used --init-extent-tree), as current checks don't report any errors.

Quoted 3 lines

The system boots and mounts the FS just fine. I can read from it all day, scrubs complete without failure.

Quoted 8 lines

but just using the system for a
while will eventually trigger the same "Transaction aborted (error -95)" error.

I realize this is something of a mess, and that I was less than methodical with my actions so far. Given that I have a full backup that can be restored if need be (and I certainly could try running the convert again), what is my best course of action?

Not a mess, I think it's a good bug report. I think Qu and David know more about the latest iteration of the convert code. If you can wait until next week at least to see if they have questions that'd be best. If you need to get access to the computer sooner than later I suggest btrfs-image -c9 -t4 -s to make a filename sanitized copy of the filesystem metadata for them to look at, just in case. They might be able to figure out the problem just from the stack trace, but better to have the image before blowing away the file system, just in case they want it.

<snip>
-95 is -EOPNOTSUPP.

Not a common errno in btrfs.

Most EOPNOTSUPP are related to discard and crapped fallcate/drop extents.

Then are you using discard mount option?

<snip>
Normally a btrfs-debug-tree would help in most case, but this time it seems to be a runtime scrub bug other than on-disk metadata corruption.

What I can see here is, with all your operation, your fs should be a normal btrfs, other than converted one.

To confirm my idea, would you please upload the following things if your filesystem is not too large?

# btrfs-debug-tree -t extent <your device>
# btrfs-debug-tree -t chunk <your device>
# btrfs-debug-tree -t dev <your device>

There is no file/dir name/data contained in the dump. So it's just chunk/extent allocation info.
You could upload them at ease.

Quoted 9 lines

Not a mess, I think it's a good bug report. I think Qu and David know more about the latest iteration of the convert code. If you can wait until next week at least to see if they have questions that'd be best. If you need to get access to the computer sooner than later I suggest btrfs-image -c9 -t4 -s to make a filename sanitized copy of the filesystem metadata for them to look at, just in case. They might be able to figure out the problem just from the stack trace, but better to have the image before blowing away the file system, just in case they want it.

Yes, btrfs-image dump would be the best.
Although sanitizing may takes a long time and the output may be too large.

Quoted 8 lines · Qu Wenruo

<snip>
-95 is -EOPNOTSUPP.

Not a common errno in btrfs.

Most EOPNOTSUPP are related to discard and crapped fallcate/drop extents.

Then are you using discard mount option?

I did indeed have the discard mount option enabled. I tried booting with discard disabled, but the same problem appeared.

Quoted 28 lines

<snip>
Normally a btrfs-debug-tree would help in most case, but this time it seems to be a runtime scrub bug other than on-disk metadata corruption.

What I can see here is, with all your operation, your fs should be a normal btrfs, other than converted one.

To confirm my idea, would you please upload the following things if your filesystem is not too large?

# btrfs-debug-tree -t extent <your device>
# btrfs-debug-tree -t chunk <your device>
# btrfs-debug-tree -t dev <your device>

There is no file/dir name/data contained in the dump. So it's just chunk/extent allocation info.
You could upload them at ease.

Quoted 9 lines

Not a mess, I think it's a good bug report. I think Qu and David know more about the latest iteration of the convert code. If you can wait until next week at least to see if they have questions that'd be best. If you need to get access to the computer sooner than later I suggest btrfs-image -c9 -t4 -s to make a filename sanitized copy of the filesystem metadata for them to look at, just in case. They might be able to figure out the problem just from the stack trace, but better to have the image before blowing away the file system, just in case they want it.

Quoted 2 lines

Yes, btrfs-image dump would be the best.
Although sanitizing may takes a long time and the output may be too large.

I had posted a btrfs-image before. It was run with a single -s flag:

http://phead.us/tmp/sgreenslade_home_sanitized_2016-09-16.btrfs

Here's the debug tree data:

http://phead.us/tmp/wheatley_chunk_2016-09-18.dump.gz
http://phead.us/tmp/wheatley_extent_2016-09-18.dump.gz
http://phead.us/tmp/wheatley_dev_2016-09-18.dump.gz

Thanks,

--Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

All chunks are completed convert to DUP, no small chunk, all to its maximum chunk size.
So from chunk level, nothing related to convert yet.

But for extent tree, I found several extents are heavily referred to. Like extent 158173081600 or 183996522496.

If you're not using off-band dedupe, then it's quite possible that's the remaining structure of convert.

Not pretty sure if it's related to the bug, but did you do the balance/defrag operation just after removing ext_save subvolume?

Quoted 9 lines · Qu Wenruo

All chunks are completed convert to DUP, no small chunk, all to its maximum chunk size.
So from chunk level, nothing related to convert yet.

But for extent tree, I found several extents are heavily referred to. Like extent 158173081600 or 183996522496.

If you're not using off-band dedupe, then it's quite possible that's the remaining structure of convert.

I never ran any sort of dedup on this partition.

Quoted 2 lines

Not pretty sure if it's related to the bug, but did you do the balance/defrag operation just after removing ext_save subvolume?

That's quite possible. I did it in a live boot, so I don't have the bash history to check. I checked it just now using "btrfs subvol list -d", and there's nothing listed. I ran a full balance after that, but the problem remains. So whatever the problem is, it can survive a full balance after the ext_save subvol is completely deleted.

--Sean

OK, I see the problem now.

The new convert is designed to create minimal number of extents, so it result the following file extents layout:

Ext2_save/image

        |
/-------------------------------\
|           Extent A             |
|<---Old Ext3 Used space  ------>|<--- Free space--->|
      \---/\---/\---/
        |    |    |
       F1    F2   F3

This causes a large extent A, refered by ext2_save/image, and files like F1/2/3 just refer to part of the large extent A.

After removing the whole ext2_save subvolume, Extent A is still there, since F1/2/3 are still referring it.

|<---Old Ext3 Used space  ------>|<--- Free space--->|
|           Extent A             |
      \---/\---/\---/
        |    |    |
       F1    F2   F3

In that case, not balance but defrag is responsible to "split" the large extent and free the unused space.

However btrfs defrag doesn't work for case like subvolume or reflink for a long time, which leaves the extent layout unchanged.
And balance doesn't handle it, as balance just relocated the large extent A and modify all referencers' pointer.

Sigh, I just forgot the fact that defrag doesn't work for a long time when designing the new convert.

So, we still need a working kernel defrag to make the fs to be a "native" btrfs.

However, such layout is completely valid for btrfs, one can generate it quite easily which following script:
----
xfs_io -f -c "pwrite 0 16M" $mnt/orig
for i in $(seq 0 15); do

	xfs_io -f -c "reflink $mnt/orig ${i}M 0 1M" $mnt/file${i}

done
----

So there is still something wrong in your backtrace so that we need to dig further.

Any idea of your load pattern to trigger the bug?

Quoted 55 lines · Qu Wenruo

OK, I see the problem now.

The new convert is designed to create minimal number of extents, so it result the following file extents layout:

Ext2_save/image

        |
/-------------------------------\
|           Extent A             |
|<---Old Ext3 Used space  ------>|<--- Free space--->|
      \---/\---/\---/
        |    |    |
       F1    F2   F3

This causes a large extent A, refered by ext2_save/image, and files like F1/2/3 just refer to part of the large extent A.

After removing the whole ext2_save subvolume, Extent A is still there, since F1/2/3 are still referring it.

|<---Old Ext3 Used space  ------>|<--- Free space--->|
|           Extent A             |
      \---/\---/\---/
        |    |    |
       F1    F2   F3

In that case, not balance but defrag is responsible to "split" the large extent and free the unused space.

However btrfs defrag doesn't work for case like subvolume or reflink for a long time, which leaves the extent layout unchanged.
And balance doesn't handle it, as balance just relocated the large extent A and modify all referencers' pointer.

Sigh, I just forgot the fact that defrag doesn't work for a long time when designing the new convert.

So, we still need a working kernel defrag to make the fs to be a "native" btrfs.

However, such layout is completely valid for btrfs, one can generate it quite easily which following script:
----
xfs_io -f -c "pwrite 0 16M" $mnt/orig
for i in $(seq 0 15); do

	xfs_io -f -c "reflink $mnt/orig ${i}M 0 1M" $mnt/file${i}

done
----

So there is still something wrong in your backtrace so that we need to dig further.

Quoted 1 line

Any idea of your load pattern to trigger the bug?

Glad to hear you've found the core of the issue.

At this point, I can trigger it immediately. As soon as I log in and run dmenu, it will attempt to rebuild its cache file (small text file that's just a list of all executables in the PATH). Once that write happens, the bug triggers and the fs goes read only.

--Sean

Quoted 6 lines

Glad to hear you've found the core of the issue.

At this point, I can trigger it immediately. As soon as I log in and run dmenu, it will attempt to rebuild its cache file (small text file that's just a list of all executables in the PATH). Once that write happens, the bug triggers and the fs goes read only.

Rewrite? Or write into new inode?

And is the same inode always causing the problem?

Quoted 10 lines · Qu Wenruo

Quoted 6 lines

Glad to hear you've found the core of the issue.

At this point, I can trigger it immediately. As soon as I log in and run dmenu, it will attempt to rebuild its cache file (small text file that's just a list of all executables in the PATH). Once that write happens, the bug triggers and the fs goes read only.

Rewrite? Or write into new inode?

And is the same inode always causing the problem?

It's not always the same. It seems like whatever triggers a write first is what kills it. I went to test it, and this time it triggered on my .bash_history file. I have bash set up with "history -a", so presumably that was an append, not an overwrite.

To cut down on the number of variables, I booted my system with the "rescue" systemd target, then su'd to my user. Simply running a few commands (with the history -a writes that bash triggered) was enough to trigger the bug. This is on 4.8.0-rc6, with the following compile time options enabled:

CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y
CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y

If I run the stock Arch kernel (4.7.2 at the moment), the issue still appears, but it takes longer. My most reliable trigger is Firefox, whose constant DB writes will trigger it within minutes.

Quoted 27 lines · Sean Greenslade

Quoted 10 lines · Qu Wenruo

Quoted 6 lines

Glad to hear you've found the core of the issue.

At this point, I can trigger it immediately. As soon as I log in and run dmenu, it will attempt to rebuild its cache file (small text file that's just a list of all executables in the PATH). Once that write happens, the bug triggers and the fs goes read only.

Rewrite? Or write into new inode?

And is the same inode always causing the problem?

It's not always the same. It seems like whatever triggers a write first is what kills it. I went to test it, and this time it triggered on my .bash_history file. I have bash set up with "history -a", so presumably that was an append, not an overwrite.

To cut down on the number of variables, I booted my system with the "rescue" systemd target, then su'd to my user. Simply running a few commands (with the history -a writes that bash triggered) was enough to trigger the bug. This is on 4.8.0-rc6, with the following compile time options enabled:

CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y
CONFIG_BTRFS_DEBUG=y
CONFIG_BTRFS_ASSERT=y

Quoted 3 lines

If I run the stock Arch kernel (4.7.2 at the moment), the issue still appears, but it takes longer. My most reliable trigger is Firefox, whose constant DB writes will trigger it within minutes.

Is there anything I can do to help this along? I can build experimental patches, set up long running scripts, run tests, whatever is necessary.

Thanks,

--Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi, all. I've been playing around with an old laptop of mine, and I figured I'd use it as a learning / bugfinding opportunity. Its /home partition was originally ext3. I have a full partition image of this drive as a backup, so I can do (and have done) potentially destructive things. The system disk is a ~6 year old SSD.

To start, I rebooted to a livedisk (Arch, kernel 4.7.2 w/progs 4.7.1) and ran a simple btrfs-convert on it. After patching up the fstab and rebooting, everything seemed fine. I deleted the recovery subvol, ran a full balance, ran a full defrag, and rebooted again. I then decided to try (as an experiment) using DUP mode for data and metadata. I ran that balance without issue, then started using the machine. Sometime later, I got the following remount ro:

[ 7316.764235] ------------[ cut here ]------------
[ 7316.764292] WARNING: CPU: 2 PID: 14196 at fs/btrfs/inode.c:2954 btrfs_finish_ordered_io+0x6bc/0x6d0 [btrfs]
[ 7316.764297] BTRFS: Transaction aborted (error -95)
[ 7316.764301] Modules linked in: fuse sha256_ssse3 sha256_generic hmac drbg ansi_cprng ctr ccm joydev mousedev uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media crc32c_generic iTCO_wdt btrfs iTCO_vendor_support arc4 xor ath9k raid6_pq ath9k_common ath9k_hw ath mac80211 snd_hda_codec_realtek snd_hda_codec_generic psmouse input_leds coretemp snd_hda_intel led_class pcspkr snd_hda_codec cfg80211 snd_hwdep snd_hda_core snd_pcm lpc_ich snd_timer atl1c rfkill snd soundcore shpchp intel_agp wmi thermal fjes battery evdev ac tpm_tis mac_hid tpm sch_fq_codel vboxnetflt(O) vboxnetadp(O) pci_stub vboxpci(O) vboxdrv(O) loop sg acpi_cpufreq ip_tables x_tables ext4 crc16 jbd2 mbcache sd_mod serio_raw atkbd libps2 ahci libahci uhci_hcd libata scsi_mod ehci_pci ehci_hcd usbcore
[ 7316.764434]  usb_common i8042 serio i915 video button intel_gtt i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm
[ 7316.764462] CPU: 2 PID: 14196 Comm: kworker/u8:11 Tainted: G           O    4.7.3-5-ck #1
[ 7316.764467] Hardware name: ASUSTeK Computer INC. 1015PEM/1015PE, BIOS 0903    11/08/2010
[ 7316.764507] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
[ 7316.764513]  0000000000000286 000000006101f47d ffff8800230dbc78 ffffffff812f0215
[ 7316.764522]  ffff8800230dbcc8 0000000000000000 ffff8800230dbcb8 ffffffff8107ae6f
[ 7316.764530]  00000b8a00000035 ffff88007791afa8 ffff8800751d9000 ffff880014101d40
[ 7316.764538] Call Trace:
[ 7316.764551]  [<ffffffff812f0215>] dump_stack+0x63/0x8e
[ 7316.764560]  [<ffffffff8107ae6f>] __warn+0xcf/0xf0
[ 7316.764567]  [<ffffffff8107aef1>] warn_slowpath_fmt+0x61/0x80
[ 7316.764605]  [<ffffffffa07aa362>] ? unpin_extent_cache+0xa2/0xf0 [btrfs]
[ 7316.764640]  [<ffffffffa07628e6>] ? btrfs_free_path+0x26/0x30 [btrfs]
[ 7316.764677]  [<ffffffffa079aaac>] btrfs_finish_ordered_io+0x6bc/0x6d0 [btrfs]
[ 7316.764715]  [<ffffffffa079adc5>] finish_ordered_fn+0x15/0x20 [btrfs]
[ 7316.764753]  [<ffffffffa07c5f8e>] btrfs_scrubparity_helper+0x7e/0x360 [btrfs]
[ 7316.764791]  [<ffffffffa07c62fe>] btrfs_endio_write_helper+0xe/0x10 [btrfs]
[ 7316.764799]  [<ffffffff810949bd>] process_one_work+0x1ed/0x490
[ 7316.764806]  [<ffffffff81094ca9>] worker_thread+0x49/0x500
[ 7316.764813]  [<ffffffff81094c60>] ? process_one_work+0x490/0x490
[ 7316.764820]  [<ffffffff8109ac3a>] kthread+0xda/0xf0
[ 7316.764830]  [<ffffffff815c553f>] ret_from_fork+0x1f/0x40
[ 7316.764838]  [<ffffffff8109ab60>] ? kthread_worker_fn+0x170/0x170
[ 7316.764843] ---[ end trace 90f54effc5e294b0 ]---
[ 7316.764851] BTRFS: error (device sda2) in btrfs_finish_ordered_io:2954: errno=-95 unknown
[ 7316.764859] BTRFS info (device sda2): forced readonly
[ 7316.765396] pending csums is 9437184

After seeing this, I decided to attempt a repair (confident that I could restore from backup if it failed). At the time, I was unaware of the issues with progs 4.7.1, so when I ran the check and saw all the incorrect backrefs messages, I figured that was my problem and ran the --repair. Of course, this didn't make the messages go away on subsequent checks, so I looked further and found this bug:

https://bugzilla.kernel.org/show_bug.cgi?id=155791

I updated progs to 4.7.2 and re-ran the --repair (I didn't save any of the logs from these, unfortunately). The repair seemed to work (I also used --init-extent-tree), as current checks don't report any errors.

The system boots and mounts the FS just fine. I can read from it all day, scrubs complete without failure, but just using the system for a while will eventually trigger the same "Transaction aborted (error -95)" error.

I realize this is something of a mess, and that I was less than methodical with my actions so far. Given that I have a full backup that can be restored if need be (and I certainly could try running the convert again), what is my best course of action?

Interesting, seems that we get errors from

btrfs_finish_ordered_io
insert_reserved_file_extent
__btrfs_drop_extents

And splitting an inline extent throws -95.