raid6 file system in a bad state

At some point in the last week, I had a 6-disk raid6 pool go south on me. One of the disks had a write problem, unbeknownst to me, which caused the entire pool and its subvolumes to remount read only.

When this problem occurred I was on debian jessie kernel 3.16.something. Following list advice I upgraded to the latest in jessie-backports, 4.7.5. My git clone of btrfs-progs is at commit 81f4d96f3d6368dc4e5edf7e3cb9d19bb4d00c4f

Not knowing the cause of the problem, I unmounted and attempted to remount, which failed, with the following coming from dmesg:

[308063.610960] BTRFS info (device sda): allowing degraded mounts
[308063.610972] BTRFS info (device sda): disk space caching is enabled
[308063.723461] BTRFS error (device sda): parent transid verify failed on
5752357961728 wanted 161562 found 159746
[308063.815224] BTRFS info (device sda): bdev /dev/sdh errs: wr 261, rd 1,
flush 87, corrupt 0, gen 0
[308063.849613] BTRFS error (device sda): parent transid verify failed on
5752642420736 wanted 161562 found 159786
[308063.881024] BTRFS error (device sda): parent transid verify failed on
5752472338432 wanted 161562 found 159751
[308063.940225] BTRFS error (device sda): parent transid verify failed on
5752478842880 wanted 161562 found 159752
[308063.979517] BTRFS error (device sda): parent transid verify failed on
5752543526912 wanted 161562 found 159764
[308064.012479] BTRFS error (device sda): parent transid verify failed on
5752513036288 wanted 161562 found 159764
[308064.049169] BTRFS error (device sda): parent transid verify failed on
5752642617344 wanted 161562 found 159786
[308064.080507] BTRFS error (device sda): parent transid verify failed on
5752642650112 wanted 161562 found 159786
[308064.138951] BTRFS error (device sda): parent transid verify failed on
5752610603008 wanted 161562 found 159783
[308064.164326] BTRFS error (device sda): bad tree block start
5918360357649457268 5752610603008
[308064.173752] BTRFS error (device sda): bad tree block start
5567295971165396096 5752610603008
[308064.182026] BTRFS error (device sda): failed to read block groups: -5
[308064.234174] BTRFS: open_ctree failed

/dev/sdh is the disc that had the write error

btrfs filesystem show produces this:

root@castor:~/btrfs-progs# btrfs filesystem show
Label: none  uuid: 73ed01df-fb2a-4b27-b6fc-12a57da934bd
        Total devices 6 FS bytes used 6.46TiB
        devid    1 size 2.73TiB used 1.64TiB path /dev/sda
        devid    2 size 2.73TiB used 1.64TiB path /dev/sdh
        devid    3 size 2.73TiB used 1.64TiB path /dev/sdd
        devid    4 size 2.73TiB used 1.64TiB path /dev/sdg
        devid    5 size 2.73TiB used 1.64TiB path /dev/sdf
        devid    6 size 2.73TiB used 1.64TiB path /dev/sde

I just now discovered the raid5/6 checksum bug and am hoping I haven't somehow hit that, since I haven't actually written much of anything to the discs in quite a long time (save for a few recently-ripped ISOs that must have been going there when the sdh write error happened).

While there's a lot of stuff I don't care about on the pool, I've got a lot of Blu Ray ISOs on it that I'd rather not have to re-rip if I can avoid it (the backups for those are the original discs in my movie cabinet), plus a local Debian mirror that I'd rather not have to re-sync.

btrfs restore gives this:

parent transid verify failed on 5752357961728 wanted 161562 found 159746 parent transid verify failed on 5752357961728 wanted 161562 found 159746 checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76 checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 checksum verify failed on 5752357961728 found 8582246F wanted B53BE280 bytenr mismatch, want=5752357961728, have=56504706479104 Couldn't setup extent tree
This is a dry-run, no files are going to be restored
Restoring /dev/null/BluRay-StarWars
Restoring /dev/null/BluRay-StarWars/Star Wars 4 A New Hope.iso Restoring /dev/null/BluRay-StarWars/Star Wars 6 Return of the Jedi.iso Restoring /dev/null/BluRay-StarWars/Star Wars Bonus Episodes 1 2 3.iso Restoring /dev/null/BluRay-StarWars/Star Wars 5 The Empire Strikes Back.iso Restoring /dev/null/BluRay-StarWars/Star Wars Spoofs and Documentaries.iso Restoring /dev/null/BluRay-StarWars/Star Wars Bonus Episodes 4 5 6.iso Restoring /dev/null/BluRay-StarWars/Star Wars 1 The Phantom Menace.iso Restoring /dev/null/BluRay-StarWars/Star Wars 2 Attack of the Clones.iso Restoring /dev/null/BluRay-StarWars/Star Wars 3 Revenge of the Sith.iso Found objectid=257, key=256
Done searching /BluRay-StarWars
Restoring /dev/null/BluRay-HarryPotter
Restoring /dev/null/BluRay-HarryPotter/Year 1 Harry Potter and the Sorcerers Stone.iso
Restoring /dev/null/BluRay-HarryPotter/Year 2 Harry Potter and the Chamber of Secrets.iso
Restoring /dev/null/BluRay-HarryPotter/Year 3 Harry Potter and the Prizoner of Azkaban.iso
Restoring /dev/null/BluRay-HarryPotter/Year 4 Harry Potter and the Goblet of Fire.iso
Restoring /dev/null/BluRay-HarryPotter/Year 5 Harry Potter and the Order of the Phoenix.iso
Found objectid=257, key=256
Done searching /BluRay-HarryPotter
Restoring /dev/null/BluRay-DowntonAbbey
Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 1 Disc 1.iso Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 1 Disc 2.iso Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 2 Disc 1.iso Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 2 Disc 2.iso Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 2 Disc 3.iso Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 3 Disc 1.iso Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 3 Disc 2.iso Restoring /dev/null/BluRay-DowntonAbbey/Downton Abbey Season 3 Disc 3.iso Found objectid=257, key=256
Done searching /BluRay-DowntonAbbey

And there's a lot of stuff in here in the output that I don't really care about so moving on to the end:

Restoring /dev/null/root/Software
Found objectid=304888, key=257402
Done searching /root/Software
Found objectid=257, key=256
Done searching /root
parent transid verify failed on 5752616845312 wanted 161562 found 159784 parent transid verify failed on 5752616845312 wanted 161562 found 159784 checksum verify failed on 5752616845312 found 0EB38D74 wanted 1BB07DCA checksum verify failed on 5752616845312 found B4E0DBD6 wanted DD91E4E9 checksum verify failed on 5752616845312 found B4E0DBD6 wanted DD91E4E9 bytenr mismatch, want=5752616845312, have=857463135251540499 Error reading subvolume /dev/null/BluRay-Disney: 18446744073709551611

After reading some of the suggestions, I attempted a btrfs rescue chunk-recover, which results in a SIGBUS error ~40% through the process:

root@castor:~/btrfs-progs#  gdb --args ./btrfs rescue chunk-recover -vvvv
/dev/sda
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./btrfs...done.
(gdb) r
Starting program: /root/btrfs-progs/btrfs rescue chunk-recover -vvvv
/dev/sda
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
All Devices:
        Device: id = 2, name = /dev/sdh
        Device: id = 3, name = /dev/sdd
        Device: id = 5, name = /dev/sdf
        Device: id = 6, name = /dev/sde
        Device: id = 4, name = /dev/sdg
        Device: id = 1, name = /dev/sda

[New Thread 0x7ffff6f91700 (LWP 14913)]
[New Thread 0x7ffff6790700 (LWP 14914)]
[New Thread 0x7ffff5f8f700 (LWP 14915)]
[New Thread 0x7ffff578e700 (LWP 14916)]
[New Thread 0x7ffff4f8d700 (LWP 14917)]
[New Thread 0x7fffe7fff700 (LWP 14918)]
Scanning: 449541562368 in dev0, 689038929920 in dev1, 681330704384 in dev2, 669722726400 in dev3, 681526239232 in dev4, 675649212416 in dev5[Thread 0x7ffff6f91700 (LWP 14913) exited]
Scanning: DONE in dev0, 1203854462976 in dev1, 1209906450432 in dev2, 1194740371456 in dev3, 1211076476928 in dev4, 1212511375360 in dev5 Program received signal SIGBUS, Bus error.
[Switching to Thread 0x7ffff4f8d700 (LWP 14917)] btrfs_new_block_group_record (leaf=leaf@entry=0x7fffdc0008c0, key=key@entry=0x7ffff4f8ccb0, slot=slot@entry=30) at cmds-check.c:5258
5258 rec->flags = btrfs_disk_block_group_flags(leaf, ptr);
(gdb) p leaf
p $1 = (struct extent_buffer *) 0x7fffdc0008c0
(gdb) p ptr
$2 = (struct btrfs_block_group_item *) 0x68eb1bad
(gdb) p *leaf
$3 = {cache_node = {rb_node = {__rb_parent_color = 0, rb_right = 0x0, rb_left = 0x0}, objectid = 0, start = 0, size = 0}, start = 0, dev_bytenr = 0,
len = 16384, tree = 0x0, lru = {next = 0x0, prev = 0x0}, recow = {next =
0x0, prev = 0x0}, refs = 0, flags = 0, fd = 0, data = 0x7fffdc000940 "5\f\004\n"}
(gdb) p *ptr
Cannot access memory at address 0x68eb1bad
(gdb) bt
#0 btrfs_new_block_group_record (leaf=leaf@entry=0x7fffdc0008c0, key=key@entry=0x7ffff4f8ccb0, slot=slot@entry=30) at cmds-check.c:5258 #1 0x0000000000434c2f in process_block_group_item (slot=30, key=0x7ffff4f8ccb0, leaf=0x7fffdc0008c0, bg_cache=0x7fffffffe998) at chunk-recover.c:232
#2 extract_metadata_record (rc=rc@entry=0x7fffffffe960, leaf=leaf@entry=0x7fffdc0008c0) at chunk-recover.c:717
#3 0x000000000043538c in scan_one_device (dev_scan_struct=0x6a6450) at chunk-recover.c:807
#4 0x00007ffff73450a4 in start_thread (arg=0x7ffff4f8d700) at pthread_create.c:309
#5 0x00007ffff707a62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

I was disappointed to see dev0 (which corresponds to /dev/sdh) come out as DONE because of these dmesg entries:

[232572.871164] mpt2sas_cm0: log_info(0x31080000): originator(PL),
code(0x08), sub_code(0x0000)
[232572.871185] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34
55 61 c0 00 00 01 00 00 00
[232572.871190] mpt2sas_cm0:    sas_address(0x50030480002e5946), phy(6)
[232572.871193] mpt2sas_cm0:
enclosure_logical_id(0x50030442523a2033),slot(2)
[232572.871197] mpt2sas_cm0:    handle(0x0012), ioc_status(success)(0x0000),
smid(36)
[232572.871200] mpt2sas_cm0:    request_len(131072), underflow(131072),
resid(131072)
[232572.871202] mpt2sas_cm0:    tag(1), transfer_count(0),
sc->result(0x00000000)
[232572.871205] mpt2sas_cm0:    scsi_status(check condition)(0x02),
scsi_state(autosense valid )(0x01)
[232572.871208] mpt2sas_cm0:    [sense_key,asc,ascq]: [0x03,0x11,0x00],
count(18)
[232572.871239] sd 0:0:8:0: [sdh] tag#4 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[232572.871243] sd 0:0:8:0: [sdh] tag#4 Sense Key : Medium Error [current]
[232572.871248] sd 0:0:8:0: [sdh] tag#4 Add. Sense: Unrecovered read error
[232572.871252] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34
55 61 c0 00 00 01 00 00 00
[232572.871256] blk_update_request: critical medium error, dev sdh, sector
878010816
[232578.796809] mpt2sas_cm0: log_info(0x31080000): originator(PL),
code(0x08), sub_code(0x0000)
[232578.796838] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34
55 61 f0 00 00 00 40 00 00
[232578.796845] mpt2sas_cm0:    sas_address(0x50030480002e5946), phy(6)
[232578.796850] mpt2sas_cm0:
enclosure_logical_id(0x50030442523a2033),slot(2)
[232578.796856] mpt2sas_cm0:    handle(0x0012), ioc_status(success)(0x0000),
smid(36)
[232578.796860] mpt2sas_cm0:    request_len(32768), underflow(32768),
resid(0)
[232578.796864] mpt2sas_cm0:    tag(0), transfer_count(32768),
sc->result(0x00000000)
[232578.796869] mpt2sas_cm0:    scsi_status(check condition)(0x02),
scsi_state(autosense valid )(0x01)
[232578.796874] mpt2sas_cm0:    [sense_key,asc,ascq]: [0x03,0x11,0x00],
count(18)
[232578.797129] sd 0:0:8:0: [sdh] tag#4 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[232578.797138] sd 0:0:8:0: [sdh] tag#4 Sense Key : Medium Error [current]
[232578.797146] sd 0:0:8:0: [sdh] tag#4 Add. Sense: Unrecovered read error
[232578.797154] sd 0:0:8:0: [sdh] tag#4 CDB: Read(16) 88 00 00 00 00 00 34
55 61 f0 00 00 00 40 00 00
[232578.797160] blk_update_request: critical medium error, dev sdh, sector
878010888
[232581.663794] mpt2sas_cm0: log_info(0x31080000): originator(PL),
code(0x08), sub_code(0x0000)
[232581.663823] sd 0:0:8:0: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 00 34
55 62 30 00 00 00 80 00 00
[232581.663830] mpt2sas_cm0:    sas_address(0x50030480002e5946), phy(6)
[232581.663835] mpt2sas_cm0:
enclosure_logical_id(0x50030442523a2033),slot(2)
[232581.663841] mpt2sas_cm0:    handle(0x0012), ioc_status(success)(0x0000),
smid(62)
[232581.663845] mpt2sas_cm0:    request_len(65536), underflow(65536),
resid(65536)
[232581.663849] mpt2sas_cm0:    tag(0), transfer_count(0),
sc->result(0x00000000)
[232581.663854] mpt2sas_cm0:    scsi_status(check condition)(0x02),
scsi_state(autosense valid )(0x01)
[232581.663859] mpt2sas_cm0:    [sense_key,asc,ascq]: [0x03,0x11,0x00],
count(18)
[232581.663918] sd 0:0:8:0: [sdh] tag#1 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[232581.663937] sd 0:0:8:0: [sdh] tag#1 Sense Key : Medium Error [current]
[232581.663951] sd 0:0:8:0: [sdh] tag#1 Add. Sense: Unrecovered read error
[232581.663960] sd 0:0:8:0: [sdh] tag#1 CDB: Read(16) 88 00 00 00 00 00 34
55 62 30 00 00 00 80 00 00
[232581.663967] blk_update_request: critical medium error, dev sdh, sector
878010928
[232584.622191] sd 0:0:8:0: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 34
55 62 08 00 00 00 08 00 00
[232584.622207] mpt2sas_cm0:    sas_address(0x50030480002e5946), phy(6)
[232584.622213] mpt2sas_cm0:
enclosure_logical_id(0x50030442523a2033),slot(2)
[232584.622219] mpt2sas_cm0:    handle(0x0012), ioc_status(success)(0x0000),
smid(55)
[232584.622224] mpt2sas_cm0:    request_len(4096), underflow(4096),
resid(4096)
[232584.622228] mpt2sas_cm0:    tag(0), transfer_count(0),
sc->result(0x00000000)
[232584.622233] mpt2sas_cm0:    scsi_status(check condition)(0x02),
scsi_state(autosense valid )(0x01)
[232584.622237] mpt2sas_cm0:    [sense_key,asc,ascq]: [0x03,0x11,0x00],
count(18)
[232584.622272] sd 0:0:8:0: [sdh] tag#0 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[232584.622282] sd 0:0:8:0: [sdh] tag#0 Sense Key : Medium Error [current]
[232584.622295] sd 0:0:8:0: [sdh] tag#0 Add. Sense: Unrecovered read error
[232584.622304] sd 0:0:8:0: [sdh] tag#0 CDB: Read(16) 88 00 00 00 00 00 34
55 62 08 00 00 00 08 00 00
[232584.622311] blk_update_request: critical medium error, dev sdh, sector
878010888
[232584.630625] Buffer I/O error on dev sdh, logical block 109751361, async

page read

rather than moving on but that's neither here nor there, since the disc really can't be trusted as it is.

btrfs check produces this output:

root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sda
parent transid verify failed on 5752357961728 wanted 161562 found 159746
parent transid verify failed on 5752357961728 wanted 161562 found 159746
checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
bytenr mismatch, want=5752357961728, have=56504706479104
Couldn't setup extent tree
ERROR: cannot open file system

Like I said, the vast majority of what's on this disc is either BluRay ISOs that I can re-rip, stuff I don't care about recovering, or stuff that I can always re-mirror if I have to. Given that I'm well versed in C programming, I'd much rather devote my time to working with the code to resolve whatever problem may be happening here than re-acquiring or re-ripping what's on that pool.

Since it took somewhere near an hour and a half to get to that SIGBUS in gdb, I've left my session open for anyone who may have ideas to chime in. Just let me know what information you need!

Thanks

Jason Michaelson

Can you be more specific about the write problem? What are the messages from the logs about these write problems and is that problem now fixed?

Sometimes it will be more tolerant with mount -o degraded,ro.

Each one of these complains about a read error and a different LBA is reported.

These should get fixed automatically by Btrfs since kernel 3.19. The problem is that you were using 3.16 so they were left to accumulate. 3.16 kernel needed a full balance to fix these.

You don't have to, that's Btrfs problem. What's confusing to me is that interspersed with these read errors, there are no Btrfs messages. I'm expecting Btrfs to do something about the read error if those sectors are in a Btrfs volume. Either Btrfs should say whether it's fixing the problem, in either case. But I see nothing.

Seems like there should be more to the story than just one bad drive or it'd be possible to mount minus the bad drive. Maybe sdh is spitting out spurious data. Have you tried physically removing /dev/sdh and then doing a degraded mount?

An entire syslog, journalctl, or dmesg might be helpful rather than excerpts.

That part I can't help out with.

Chris Murphy

What do you get for

btrfs-find-root <dev>
btrfs rescue super-recover -v <dev>

It shouldn't matter which dev you pick, unless it face plants, then try another.

root@castor:~/logs# btrfs-find-root /dev/sda
parent transid verify failed on 5752357961728 wanted 161562 found 159746
parent transid verify failed on 5752357961728 wanted 161562 found 159746
Couldn't setup extent tree
Superblock thinks the generation is 161562
Superblock thinks the level is 1

There's no further output, and btrfs-find-root is pegged at 100%.

At the moment, the perceived bad disc is connected. I received the same results without as well.

root@castor:~/logs# btrfs rescue super-recover -v /dev/sda
All Devices:
        Device: id = 2, name = /dev/sdh
        Device: id = 3, name = /dev/sdd
        Device: id = 5, name = /dev/sdf
        Device: id = 6, name = /dev/sde
        Device: id = 4, name = /dev/sdg
        Device: id = 1, name = /dev/sda

Before Recovering:
[All good supers]:
device name = /dev/sdd
superblock bytenr = 65536

device name = /dev/sdd
superblock bytenr = 67108864

device name = /dev/sdd
superblock bytenr = 274877906944

device name = /dev/sdf
superblock bytenr = 65536

device name = /dev/sdf
superblock bytenr = 67108864

device name = /dev/sdf
superblock bytenr = 274877906944

device name = /dev/sde
superblock bytenr = 65536

device name = /dev/sde
superblock bytenr = 67108864

device name = /dev/sde
superblock bytenr = 274877906944

device name = /dev/sdg
superblock bytenr = 65536

device name = /dev/sdg
superblock bytenr = 67108864

device name = /dev/sdg
superblock bytenr = 274877906944

device name = /dev/sda
superblock bytenr = 65536

device name = /dev/sda
superblock bytenr = 67108864

device name = /dev/sda
superblock bytenr = 274877906944

[All bad supers]:
device name = /dev/sdh
superblock bytenr = 65536

device name = /dev/sdh
superblock bytenr = 67108864

device name = /dev/sdh
superblock bytenr = 274877906944

Make sure this is a btrfs disk otherwise the tool will destroy other fs, Are you sure? [y/N]: n Aborted to recover bad superblocks

I aborted this waiting for instructions on whether to proceed from the list.

Bad superblocks can't be a good thing and would only cause confusion. I'd think that a known bad superblock would be ignored at mount time and even by btrfs-find-root, or maybe even replaced like any other kind of known bad metadata where good copies are available.

btrfs-show-super -f /dev/sda
btrfs-show-super -f /dev/sdh

Find out what the difference is between good and bad supers.

Chris Murphy

root@castor:~# btrfs-show-super -f /dev/sda
superblock: bytenr=65536, device=/dev/sda
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0x45278835 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    73ed01df-fb2a-4b27-b6fc-12a57da934bd
label
generation              161562
root                    5752616386560
sys_array_size          354
chunk_root_generation   156893
root_level              1
chunk_root              20971520
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             18003557892096
bytes_used              7107627130880
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             6
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0xe1
                        ( MIXED_BACKREF |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          RAID56 )
cache_generation        161562
uuid_tree_generation    161562
dev_item.uuid           08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4
dev_item.fsid           73ed01df-fb2a-4b27-b6fc-12a57da934bd [match]
dev_item.type           0
dev_item.total_bytes    3000592982016
dev_item.bytes_used     1800957198336
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0)
                chunk length 4194304 owner 2 stripe_len 65536
                type SYSTEM num_stripes 1
                        stripe 0 devid 1 offset 0
                        dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4
        item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
                chunk length 11010048 owner 2 stripe_len 65536
                type SYSTEM|RAID6 num_stripes 6
                        stripe 0 devid 6 offset 1048576
                        dev uuid: 390a1fd8-cc6c-40e7-b0b5-88ca7dcbcc32
                        stripe 1 devid 5 offset 1048576
                        dev uuid: 2df974c5-9dde-4062-81e9-c6eeee13db62
                        stripe 2 devid 4 offset 1048576
                        dev uuid: dce3d159-721d-4859-9955-37a03769bb0d
                        stripe 3 devid 3 offset 1048576
                        dev uuid: 6f7142db-824c-4791-a5b2-d6ce11c81c8f
                        stripe 4 devid 2 offset 1048576
                        dev uuid: dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c
                        stripe 5 devid 1 offset 20971520
                        dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4
backup_roots[4]:
        backup 0:
                backup_tree_root:       5752437456896   gen: 161561     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
                backup_extent_root:     5752385224704   gen: 161561     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
                backup_dev_root:        5752437587968   gen: 161561     level: 1
                backup_csum_root:       5752389615616   gen: 161561     level: 3
                backup_total_bytes:     18003557892096
                backup_bytes_used:      7112579833856
                backup_num_devices:     6

        backup 1:
                backup_tree_root:       5752616386560   gen: 161562     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
                backup_extent_root:     5752649416704   gen: 161563     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
                backup_dev_root:        5752616501248   gen: 161562     level: 1
                backup_csum_root:       5752650203136   gen: 161563     level: 3
                backup_total_bytes:     18003557892096
                backup_bytes_used:      7107602407424
                backup_num_devices:     6

        backup 2:
                backup_tree_root:       5752112103424   gen: 161559     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
                backup_extent_root:     5752207409152   gen: 161560     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
                backup_dev_root:        5752113463296   gen: 161559     level: 1
                backup_csum_root:       5752205492224   gen: 161560     level: 3
                backup_total_bytes:     18003557892096
                backup_bytes_used:      7112514002944
                backup_num_devices:     6

        backup 3:
                backup_tree_root:       5752298307584   gen: 161560     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
                backup_extent_root:     5752385224704   gen: 161561     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
                backup_dev_root:        5752299978752   gen: 161560     level: 1
                backup_csum_root:       5752389615616   gen: 161561     level: 3
                backup_total_bytes:     18003557892096
                backup_bytes_used:      7112542425088
                backup_num_devices:     6

root@castor:~# btrfs-show-super -f /dev/sdh
superblock: bytenr=65536, device=/dev/sdh
---------------------------------------------------------
csum_type               0 (crc32c)
csum_size               4
csum                    0x0f7dfe09 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    73ed01df-fb2a-4b27-b6fc-12a57da934bd
label
generation              161474
root                    4844272943104
sys_array_size          354
chunk_root_generation   156893
root_level              1
chunk_root              20971520
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             18003557892096
bytes_used              7110395990016
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             6
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0xe1
                        ( MIXED_BACKREF |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          RAID56 )
cache_generation        161474
uuid_tree_generation    161474
dev_item.uuid           dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c
dev_item.fsid           73ed01df-fb2a-4b27-b6fc-12a57da934bd [match]
dev_item.type           0
dev_item.total_bytes    3000592982016
dev_item.bytes_used     1800936226816
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          2
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0)
                chunk length 4194304 owner 2 stripe_len 65536
                type SYSTEM num_stripes 1
                        stripe 0 devid 1 offset 0
                        dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4
        item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
                chunk length 11010048 owner 2 stripe_len 65536
                type SYSTEM|RAID6 num_stripes 6
                        stripe 0 devid 6 offset 1048576
                        dev uuid: 390a1fd8-cc6c-40e7-b0b5-88ca7dcbcc32
                        stripe 1 devid 5 offset 1048576
                        dev uuid: 2df974c5-9dde-4062-81e9-c6eeee13db62
                        stripe 2 devid 4 offset 1048576
                        dev uuid: dce3d159-721d-4859-9955-37a03769bb0d
                        stripe 3 devid 3 offset 1048576
                        dev uuid: 6f7142db-824c-4791-a5b2-d6ce11c81c8f
                        stripe 4 devid 2 offset 1048576
                        dev uuid: dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c
                        stripe 5 devid 1 offset 20971520
                        dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4
backup_roots[4]:
        backup 0:
                backup_tree_root:       4844253364224   gen: 161473     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
                backup_extent_root:     4844248121344   gen: 161473     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
                backup_dev_root:        1411186688      gen: 156893     level: 1
                backup_csum_root:       4844247793664   gen: 161473     level: 3
                backup_total_bytes:     18003557892096
                backup_bytes_used:      7110380077056
                backup_num_devices:     6

        backup 1:
                backup_tree_root:       4844272943104   gen: 161474     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
                backup_extent_root:     4844268240896   gen: 161474     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
                backup_dev_root:        1411186688      gen: 156893     level: 1
                backup_csum_root:       4844254216192   gen: 161474     level: 3
                backup_total_bytes:     18003557892096
                backup_bytes_used:      7110395990016
                backup_num_devices:     6

        backup 2:
                backup_tree_root:       4844252168192   gen: 161471     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
                backup_extent_root:     4844242698240   gen: 161471     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
                backup_dev_root:        1411186688      gen: 156893     level: 1
                backup_csum_root:       4844241764352   gen: 161471     level: 3
                backup_total_bytes:     18003557892096
                backup_bytes_used:      7110343888896
                backup_num_devices:     6

        backup 3:
                backup_tree_root:       4844263358464   gen: 161472     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
                backup_extent_root:     4844261965824   gen: 161472     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
                backup_dev_root:        1411186688      gen: 156893     level: 1
                backup_csum_root:       4844261801984   gen: 161472     level: 3
                backup_total_bytes:     18003557892096
                backup_bytes_used:      7110370037760
                backup_num_devices:     6

root@castor:~# btrfs-show-super -f /dev/sda > sda
root@castor:~# btrfs-show-super -f /dev/sdh > sdh
root@castor:~# diff -u sda sdh
--- sda 2016-10-11 11:09:42.853170807 -0500
+++ sdh 2016-10-11 11:09:46.469082028 -0500
@@ -1,16 +1,16 @@
-superblock: bytenr=65536, device=/dev/sda
+superblock: bytenr=65536, device=/dev/sdh
 ---------------------------------------------------------
 csum_type              0 (crc32c)
 csum_size              4
-csum                   0x45278835 [match]
+csum                   0x0f7dfe09 [match]
 bytenr                 65536
 flags                  0x1
                        ( WRITTEN )
 magic                  _BHRfS_M [match]
 fsid                   73ed01df-fb2a-4b27-b6fc-12a57da934bd
 label
-generation             161562
-root                   5752616386560
+generation             161474
+root                   4844272943104
 sys_array_size         354
 chunk_root_generation  156893
 root_level             1
@@ -20,7 +20,7 @@
 log_root_transid       0
 log_root_level         0
 total_bytes            18003557892096
-bytes_used             7107627130880
+bytes_used             7110395990016
 sectorsize             4096
 nodesize               16384
 leafsize               16384
@@ -34,17 +34,17 @@
                          BIG_METADATA |
                          EXTENDED_IREF |
                          RAID56 )
-cache_generation       161562
-uuid_tree_generation   161562
-dev_item.uuid          08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4
+cache_generation       161474
+uuid_tree_generation   161474
+dev_item.uuid          dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c
 dev_item.fsid          73ed01df-fb2a-4b27-b6fc-12a57da934bd [match]
 dev_item.type          0
 dev_item.total_bytes   3000592982016
-dev_item.bytes_used    1800957198336
+dev_item.bytes_used    1800936226816
 dev_item.io_align      4096
 dev_item.io_width      4096
 dev_item.sector_size   4096
-dev_item.devid         1
+dev_item.devid         2
 dev_item.dev_group     0
 dev_item.seek_speed    0
 dev_item.bandwidth     0
@@ -72,47 +72,47 @@
                        dev uuid: 08c50aa9-c2dd-43b7-a631-6dfdc7d69ea4
 backup_roots[4]:
        backup 0:
-               backup_tree_root:       5752437456896   gen: 161561     level: 1
+               backup_tree_root:       4844253364224   gen: 161473     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
-               backup_extent_root:     5752385224704   gen: 161561     level: 2
+               backup_extent_root:     4844248121344   gen: 161473     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
-               backup_dev_root:        5752437587968   gen: 161561     level: 1
-               backup_csum_root:       5752389615616   gen: 161561     level: 3
+               backup_dev_root:        1411186688      gen: 156893     level: 1
+               backup_csum_root:       4844247793664   gen: 161473     level: 3
                backup_total_bytes:     18003557892096
-               backup_bytes_used:      7112579833856
+               backup_bytes_used:      7110380077056
                backup_num_devices:     6

backup 1:

-               backup_tree_root:       5752616386560   gen: 161562     level: 1
+               backup_tree_root:       4844272943104   gen: 161474     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
-               backup_extent_root:     5752649416704   gen: 161563     level: 2
+               backup_extent_root:     4844268240896   gen: 161474     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
-               backup_dev_root:        5752616501248   gen: 161562     level: 1
-               backup_csum_root:       5752650203136   gen: 161563     level: 3
+               backup_dev_root:        1411186688      gen: 156893     level: 1
+               backup_csum_root:       4844254216192   gen: 161474     level: 3
                backup_total_bytes:     18003557892096
-               backup_bytes_used:      7107602407424
+               backup_bytes_used:      7110395990016
                backup_num_devices:     6

backup 2:

-               backup_tree_root:       5752112103424   gen: 161559     level: 1
+               backup_tree_root:       4844252168192   gen: 161471     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
-               backup_extent_root:     5752207409152   gen: 161560     level: 2
+               backup_extent_root:     4844242698240   gen: 161471     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
-               backup_dev_root:        5752113463296   gen: 161559     level: 1
-               backup_csum_root:       5752205492224   gen: 161560     level: 3
+               backup_dev_root:        1411186688      gen: 156893     level: 1
+               backup_csum_root:       4844241764352   gen: 161471     level: 3
                backup_total_bytes:     18003557892096
-               backup_bytes_used:      7112514002944
+               backup_bytes_used:      7110343888896
                backup_num_devices:     6

backup 3:

-               backup_tree_root:       5752298307584   gen: 161560     level: 1
+               backup_tree_root:       4844263358464   gen: 161472     level: 1
                backup_chunk_root:      20971520        gen: 156893     level: 1
-               backup_extent_root:     5752385224704   gen: 161561     level: 2
+               backup_extent_root:     4844261965824   gen: 161472     level: 2
                backup_fs_root:         124387328       gen: 74008      level: 0
-               backup_dev_root:        5752299978752   gen: 161560     level: 1
-               backup_csum_root:       5752389615616   gen: 161561     level: 3
+               backup_dev_root:        1411186688      gen: 156893     level: 1
+               backup_csum_root:       4844261801984   gen: 161472     level: 3
                backup_total_bytes:     18003557892096
-               backup_bytes_used:      7112542425088
+               backup_bytes_used:      7110370037760
                backup_num_devices:     6

OK so most obvious is that the bad super is many generations back than the good super. That's expected given all the write errors.

This squares with the good super. So btrfs-find-root is using a good super. I don't know what 5752357961728 is for, maybe it's possible to read that with btrfs-debug-tree -b 5752357961728 <anydev> and see what comes back. This is not the tree root according to the super though. So what do you get for btrfs-debug-tree -b 5752616386560 <anydev>

Going back to your logs....

[   38.810575] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state
recovery directory
[   38.810595] NFSD: starting 90-second grace period (net ffffffffb12e5b80)
[  241.292816] INFO: task bfad_worker:234 blocked for more than 120 seconds.
[  241.299135]       Not tainted 4.7.0-0.bpo.1-amd64 #1
[  241.305645] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"

disables this message.

I don't know what this kernel is. I think you'd be better off with stable 4.7.7 or 4.8.1 for this work, so you're not running into a bunch of weird blocked task problems in addition to whatever is going on with the fs.

[   38.810575] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state
recovery directory
[   38.810595] NFSD: starting 90-second grace period (net ffffffffb12e5b80)
[  241.292816] INFO: task bfad_worker:234 blocked for more than 120 seconds.
[  241.299135]       Not tainted 4.7.0-0.bpo.1-amd64 #1
[  241.305645] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"

disables this message.

I don't know what this kernel is. I think you'd be better off with stable 4.7.7 or 4.8.1 for this work, so you're not running into a bunch of weird blocked task problems in addition to whatever is going on with the fs.

[   20.552205] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd
devid 3 transid 161562 /dev/sdd
[   20.552372] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd
devid 5 transid 161562 /dev/sdf
[   20.552524] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd
devid 6 transid 161562 /dev/sde
[   20.552689] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd
devid 4 transid 161562 /dev/sdg
[   20.552858] BTRFS: device fsid 73ed01df-fb2a-4b27-b6fc-12a57da934bd
devid 1 transid 161562 /dev/sda
[  669.843166] BTRFS warning (device sda): devid 2 uuid
dc8760f1-2c54-4134-a9a7-a0ac2b7a9f1c is missing
[232572.871243] sd 0:0:8:0: [sdh] tag#4 Sense Key : Medium Error [current]

Two items missing, in effect, for this failed read. One literally missing, and the other one missing due to unrecoverable read error. The fact it's not trying to fix anything suggests it hasn't really finished mounting, there must be something wrong where it either just gets confused and won't fix (because it might make things worse) or there isn't reduncancy.

[52799.495999] mce: [Hardware Error]: Machine check events logged
[53249.491975] mce: [Hardware Error]: Machine check events logged
[231298.005594] mce: [Hardware Error]: Machine check events logged

Bunch of other hardware issues...

I *really* think you need to get the hardware issues sorted out before working on this file system unless you just don't care that much about it. There are already enough unknowns without contributing who knows what effect the hardware issues are having while trying to repair things. Or even understand what's going on.

Huh, well item 0 is damn strange. I wonder how that happened. The dev uuid of that single system chunk matches devid 1. This is a single source of failure. This could be an artifact of creating the raid6 with an old btrfs-progs. I just created a volume with btrfs-progs 4.7.3:

# mkfs.btrfs -draid6 -mraid6 /dev/mapper/VG-1 /dev/mapper/VG-2
/dev/mapper/VG-3 /dev/mapper/VG-4 /dev/mapper/VG-5 /dev/mapper/VG-6

And the super block sys_chunk_array has only a SYSTEM|RAID6 chunk. There is no single SYSTEM chunk. After mounting and copying some data over, then umounting, same thing. One system chunk, raid6.

So *IF* there is anything wrong with this single system chunk, it's all bets are off, no way to even attempt to fix the problem. That might explain why it's not getting past the very earliest stage of mounting. But it's inconclusive.

Chris Murphy

readding btrfs

Yes with -t option and that root bytenr for the generation you want to restore. Thing is, that's so far back the metadata may be gone (overwritten) already. But worth a shot. I've recovered recently deleted files this way.

OK at this point I'm thinking that fixing the super blocks won't change anything because it sounds like it's using the new ones anyway and maybe the thing to try is going back to a tree root that isn't in any of the new supers. That means losing anything that was being written when the lost writes happened. However, for all we know some overwrites happened so this won't work. And also it does nothing to deal with the fragile state of having at least two flaky devices, and one of the system chunks with no redundancy.

Try 'btrfs check' without repair. And then also try it with -r flag using the various tree roots we've seen so far. Try explicitly using 5752616386560, which is what it ought to use first anyway. And then also 4844272943104.

That might go far enough back before the bad sectors were a factor. Normally what you'd want is for it to use one of the backup roots, but it's consistently running into a problem with all of them when using recovery mount option.

Chris Murphy

With the bad disc in place:

root@castor:~/btrfs-progs# ./btrfs restore -t 4844272943104 -D  /dev/sda /dev/null
parent transid verify failed on 4844272943104 wanted 161562 found 161476
parent transid verify failed on 4844272943104 wanted 161562 found 161476
checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E
checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640
checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640
bytenr mismatch, want=4844272943104, have=66211125067776
Couldn't read tree root
Could not open root, trying backup super
warning, device 6 is missing
warning, device 5 is missing
warning, device 4 is missing
warning, device 3 is missing
warning, device 2 is missing
checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB
checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB
bytenr mismatch, want=20971520, have=267714560
ERROR: cannot read chunk root
Could not open root, trying backup super
warning, device 6 is missing
warning, device 5 is missing
warning, device 4 is missing
warning, device 3 is missing
warning, device 2 is missing
checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB
checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB
bytenr mismatch, want=20971520, have=267714560
ERROR: cannot read chunk root
Could not open root, trying backup super

And what's interesting is that when I move the /dev/sdd (the current bad disc) out of /dev, rescan, and run btrfs restore with the main root I get similar output:

root@castor:~/btrfs-progs# ./btrfs restore -D  /dev/sda /dev/null
warning, device 2 is missing
checksum verify failed on 21430272 found 71001E6E wanted 95E3A3D8
checksum verify failed on 21430272 found 992E0C37 wanted 36992D8B
checksum verify failed on 21430272 found 992E0C37 wanted 36992D8B
bytenr mismatch, want=21430272, have=264830976
Couldn't read chunk tree
Could not open root, trying backup super
warning, device 6 is missing
warning, device 5 is missing
warning, device 4 is missing
warning, device 3 is missing
warning, device 2 is missing
checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB
checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB
bytenr mismatch, want=20971520, have=267714560
ERROR: cannot read chunk root
Could not open root, trying backup super
warning, device 6 is missing
warning, device 5 is missing
warning, device 4 is missing
warning, device 3 is missing
warning, device 2 is missing
checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB
checksum verify failed on 20971520 found 0FBD46D5 wanted FC3EB3AB
bytenr mismatch, want=20971520, have=267714560
ERROR: cannot read chunk root
Could not open root, trying backup super

So it doesn't seem to work, but the difference in output between the two, at least to my untrained eyes, is intriguing, to say the least.

This is the one thing I'm not following you on. I know there's one device that's flaky. Originally sdi, switched to sdh, and today (after reboot to 4.7.7), sdd. You'll have to forgive my ignorance, but I'm missing how you determined that a second was flaky (or was that from the ITEM 0 not being replicated you mentioned yesterday?)

root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sda
parent transid verify failed on 5752357961728 wanted 161562 found 159746
parent transid verify failed on 5752357961728 wanted 161562 found 159746
checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
bytenr mismatch, want=5752357961728, have=56504706479104
Couldn't setup extent tree
ERROR: cannot open file system
root@castor:~/btrfs-progs# ./btrfs check --readonly /dev/sdd
parent transid verify failed on 4844272943104 wanted 161474 found 161476
parent transid verify failed on 4844272943104 wanted 161474 found 161476
checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E
checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640
checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640
bytenr mismatch, want=4844272943104, have=66211125067776
Couldn't read tree root
ERROR: cannot open file system

root@castor:~/btrfs-progs# ./btrfs check --readonly -r 5752616386560 /dev/sda
parent transid verify failed on 5752357961728 wanted 161562 found 159746
parent transid verify failed on 5752357961728 wanted 161562 found 159746
checksum verify failed on 5752357961728 found B5CA97C0 wanted 51292A76
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
checksum verify failed on 5752357961728 found 8582246F wanted B53BE280
bytenr mismatch, want=5752357961728, have=56504706479104
Couldn't setup extent tree
ERROR: cannot open file system
root@castor:~/btrfs-progs# ./btrfs check --readonly -r 5752616386560 /dev/sdd
parent transid verify failed on 5752616386560 wanted 161474 found 161562
parent transid verify failed on 5752616386560 wanted 161474 found 161562
checksum verify failed on 5752616386560 found 2A134884 wanted CEF0F532
checksum verify failed on 5752616386560 found B7FE62DB wanted 3786D60F
checksum verify failed on 5752616386560 found B7FE62DB wanted 3786D60F
bytenr mismatch, want=5752616386560, have=56504661311488
Couldn't read tree root
ERROR: cannot open file system

root@castor:~/btrfs-progs# ./btrfs check --readonly -r 4844272943104 /dev/sda
parent transid verify failed on 4844272943104 wanted 161562 found 161476
parent transid verify failed on 4844272943104 wanted 161562 found 161476
checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E
checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640
checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640
bytenr mismatch, want=4844272943104, have=66211125067776
Couldn't read tree root
ERROR: cannot open file system
root@castor:~/btrfs-progs# ./btrfs check --readonly -r 4844272943104 /dev/sdd
parent transid verify failed on 4844272943104 wanted 161474 found 161476
parent transid verify failed on 4844272943104 wanted 161474 found 161476
checksum verify failed on 4844272943104 found E808AB28 wanted 0CEB169E
checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640
checksum verify failed on 4844272943104 found 4694222D wanted 5D4F0640
bytenr mismatch, want=4844272943104, have=66211125067776
Couldn't read tree root
ERROR: cannot open file system

Is that a result of all of them being identical, save for the bad disc?

Again, Chris, Thank you so much for your time looking at this! Btrfs on the whole is something that, as a developer, I'd love to become involved with. Alas, there are only 24 hours in the day.

Don't all of these device missing messages seem bogus? I don't know how to find out what's going on here. If it were me, I'd try to reproduce this with a couple of distros's live images (Fedora Rawhide and openSUSE Tumbleweed), and if they're both reproducing this "missing" output, I'd file a bugzilla.kernel.org bug with a strace. I mean, this stuff is hard enough as it is without bugs like this getting in the way.

Fedora 25 nightly:
https://kojipkgs.fedoraproject.org/compose/branched/Fedora-25-20161008.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-25-20161008.n.0.iso

That'll have some version of kernel 4.8, not sure which one. And it will have btrfs-progs 4.6.1 which is safe but showing its age in Btrfs years.

You can do this from inside the live environment:

sudo dnf update
https://kojipkgs.fedoraproject.org//packages/btrfs-progs/4.7.3/1.fc26/x86_64/btrfs-progs-4.7.3-1.fc26.x86_64.rpm

or

sudo dnf update
https://kojipkgs.fedoraproject.org//packages/btrfs-progs/4.8.1/1.fc26/x86_64/btrfs-progs-4.8.1-1.fc26.x86_64.rpm

It's probably just as valid to do this with whatever you have now, strace that and file a bug. But it doesn't really for sure isolate whether it's a local problem or not.

Yeah I'm not sure what to recommend now.

In your dmesg there was one device reported missing entirely, and then a separate device had a sector read failure.

Someone else who knows more will have to speak up. This is one of the more annoying things about Btrfs's state right now, is it's not at all clear to a regular user what sequence to attempt repairs in. It's a shot in the dark. Other file systems it's much easier. It fails to mount, you run fsck with default options, and it either can fix it or it can't. Btrfs, it's many options, many orders, very developer oriented messages, and no hints what the next step is to take.

At this point you could set up some kind of overlay on each drive, maybe also using blockdev to set each block device read only to ensure the original is not modified.

Something like this:
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file

But that will avoid this: "Block-level copies of devices"
https://btrfs.wiki.kernel.org/index.php/Gotchas

I haven't tried this, so I'm not really certain how to hide the original and overlay from the kernel since they both have to be present at the same time. Maybe an LVM snapshot LV can be presented to libvirt/virt-manager and you can use one a recent distro image to boot the VM and try some repairs. I just can't tell you what order to do them in.

Cannot read chunk root is a problem, maybe it can be repaired with btrfs rescue chunk-recover. Cannot read tree root is also a problem, once the chunk is repaired, maybe it's possible to repair it. The extent tree can't be used until the chunk tree is readable so that ought to just take care of itself. You might be looking at chunk recover, super recover, check --repair, and maybe even check --repair --init-extent-tree. And as a last resort --init-csum-tree which really is just papering over real problems in a way that now the file system won't know what's bad and makes things worse but it might survive long enough to get more data off.

And actually, before any of the above, you could see if you can take a btrfs-image -t4 -c9 -s, and also btrfs-debug-tree and output to a file somewhere. Maybe then it's a useful donation image for making the tools better.

I don't understand the question. The bad disk is the one that has the bad super, but all the tools are clearly ignoring the bad super when looking for the tree root. So I don't think the bad disk is a factor. I can't prove it but I think the problems were happening before the bad disk, it's just that the bad disk added to the confusion and may also be preventing repairs.

Chris Murphy

This may be relevant and is pretty terrible.

http://www.spinics.net/lists/linux-btrfs/msg59741.html

Chris Murphy

I've been following that thread. It's been my fear.

I'm in the process of doing a restore of what I can get off of it so that i can re-create the file system with raid1 which, if i'm reading that thread correctly doesn't suffer at all from the rmw problems extant in the raid5/6 code at the moment.

Again, thanks for your help.