Skip to contentSkip to conversation

recovering from "parent transid verify failed"

DISCUSSION Tim Walberg 2 participants 8 messages Aug 14, 2019 – Aug 15, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Most of the recommendations I've found online deal with when "wanted" is greater than "found", which, if I understand correctly means that one or more transactions were interrupted/lost before fully committed.

Are the recommendations for recovery the same if the system is reporting a "wanted" that is less than "found"?

Signature · 1 line
[email protected]
#1 lore raw
Quoted 3 lines · Tim Walberg

Most of the recommendations I've found online deal with when "wanted" is greater than "found", which, if I understand correctly means that one or more transactions were interrupted/lost before fully committed.

No matter what the case is, a proper transaction shouldn't have any tree block overwritten.

That means, either the FLUSH/FUA of the hardware/lower block layer is screwed up, or the COW of tree block is already screwed up.

Quoted 4 lines

Are the recommendations for recovery the same if the system is reporting a "wanted" that is less than "found"?

The salvage is no difference than any transid mismatch, no matter if it's larger or smaller.

It depends on the tree block.

Please provide full dmesg output and btrfs check for further advice.

Thanks,
Qu

#2 lore raw

Had to wait for 'btrfs recover' to finish before I proceed farther.

Kernel is 4.19.45, tools are 4.19.1

File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB)

Output from attempting to mount:

# mount -o ro,usebackuproot /dev/sdc1 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
       missing codepage or helper program, or other error

In some cases useful info is found in syslog - try dmesg | tail or so.

Kernel messages from the mount attempt:

[Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time
[Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled
[Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents
[Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750
[Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750
[Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5
[Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed

Output from 'btrfs check -p /dev/sdc1':

# btrfs check -p /dev/sdc1
Opening filesystem to check...
parent transid verify failed on 229846466560 wanted 49749 found 49750
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2
ERROR: cannot open file system
Quoted 27 lines · Qu Wenruo
Quoted 27 lines · Tim Walberg

> Most of the recommendations I've found online deal with when "wanted" is > greater than "found", which, if I understand correctly means that one or > more transactions were interrupted/lost before fully committed.

No matter what the case is, a proper transaction shouldn't have any tree block overwritten.

That means, either the FLUSH/FUA of the hardware/lower block layer is screwed up, or the COW of tree block is already screwed up.

>
> Are the recommendations for recovery the same if the system is reporting a > "wanted" that is less than "found"?
>
The salvage is no difference than any transid mismatch, no matter if it's larger or smaller.

It depends on the tree block.

Please provide full dmesg output and btrfs check for further advice.

Thanks,
Qu

Signature · 6 lines
+----------------------+
| Tim Walberg          |
| 830 Carriage Dr.     |
| Algonquin, IL 60102  |
| [email protected] |
+----------------------+
#7 lore raw
In reply to Qu Wenruo

Also - here's 'btrfs inspect-internal dump-super /dev/sdc1':

superblock: bytenr=65536, device=/dev/sdc1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x4331039b [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0

total_bytes   9001775738880
bytes_used    2085801975808
sectorsize    4096

nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid 7338a973-9a45-4032-a4c9-d18142fd7908
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1407675531264
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0

Although, 'btrfs inspect-internal logical-resolve ...' just says:

# btrfs inspect-internal logical-resolve 229846466560 /dev/sdc1
ERROR: not a btrfs filesystem: /dev/sdc1
Quoted 20 lines · Qu Wenruo
Quoted 20 lines · Tim Walberg

> Most of the recommendations I've found online deal with when "wanted" is > greater than "found", which, if I understand correctly means that one or > more transactions were interrupted/lost before fully committed.

No matter what the case is, a proper transaction shouldn't have any tree block overwritten.

That means, either the FLUSH/FUA of the hardware/lower block layer is screwed up, or the COW of tree block is already screwed up.

>
> Are the recommendations for recovery the same if the system is reporting a > "wanted" that is less than "found"?
>
The salvage is no difference than any transid mismatch, no matter if it's larger or smaller.

Quoted 7 lines
Quoted 7 lines

It depends on the tree block.

Please provide full dmesg output and btrfs check for further advice.

Thanks,
Qu

End of included message

Signature · 6 lines
+----------------------+
| Tim Walberg          |
| 830 Carriage Dr.     |
| Algonquin, IL 60102  |
| [email protected] |
+----------------------+
#3 lore raw
In reply to Tim Walberg
Quoted 15 lines · Tim Walberg

Had to wait for 'btrfs recover' to finish before I proceed farther.

Kernel is 4.19.45, tools are 4.19.1

File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB)

Output from attempting to mount:

# mount -o ro,usebackuproot /dev/sdc1 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
       missing codepage or helper program, or other error

In some cases useful info is found in syslog - try dmesg | tail or so.

Quoted 8 lines

Kernel messages from the mount attempt:

[Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time
[Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled
[Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents
[Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750
[Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750
[Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5

Extent tree corruption.

So if that's the only corruption, you have a very high chance to recover most of your data.

Btrfs rescue can work, or you can try the experimental patches which provides rescue=skip_bg mount option to allow you mount the fs RO and receive your data (later is way faster than user space rescue)
https://patchwork.kernel.org/project/linux-btrfs/list/?series=130637

Also, for your dump super output, it doesn't provide too much info.

You would like to use -Ffa option for more info.
Also, you could also try that on all 3 devices, to find out which one has lower generation.

Also, please provide the history of the corruption.
One generation corruptions is a little rare. Is sudden power loss involved in this case?

Thanks,
Qu

Quoted 45 lines
[Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed

Output from 'btrfs check -p /dev/sdc1':

# btrfs check -p /dev/sdc1
Opening filesystem to check...
parent transid verify failed on 229846466560 wanted 49749 found 49750
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2
ERROR: cannot open file system
Quoted 27 lines · Qu Wenruo
Quoted 27 lines · Tim Walberg

> Most of the recommendations I've found online deal with when "wanted" is > greater than "found", which, if I understand correctly means that one or > more transactions were interrupted/lost before fully committed.

No matter what the case is, a proper transaction shouldn't have any tree block overwritten.

That means, either the FLUSH/FUA of the hardware/lower block layer is screwed up, or the COW of tree block is already screwed up.

>
> Are the recommendations for recovery the same if the system is reporting a > "wanted" that is less than "found"?
>
The salvage is no difference than any transid mismatch, no matter if it's larger or smaller.

It depends on the tree block.

Please provide full dmesg output and btrfs check for further advice.

Thanks,
Qu

#4 lore raw

'dump-super -Ffa' from all three devices attached.

'btrfs restore' did appear to recover most of the main data, minus snapshots, which would have greatly increased the required time and capacity, since I was recovering to XFS.

'btrfs rescue chunk-recover' ran, but failed to fix anything. 'btrfs rescue super-recover' says all supers are fine.

Initial corruption was due to a hard hang, which didn't leave enough crumbs to determine the source - might have been btrfs, might have been nvidia, might have been something completely different.

Quoted 88 lines · Qu Wenruo
Quoted 88 lines · Tim Walberg

> Had to wait for 'btrfs recover' to finish before I proceed farther.
>
> Kernel is 4.19.45, tools are 4.19.1
>
> File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB)
>
> Output from attempting to mount:
>
> # mount -o ro,usebackuproot /dev/sdc1 /mnt
> mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
> missing codepage or helper program, or other error
>
> In some cases useful info is found in syslog - try
> dmesg | tail or so.
>
> Kernel messages from the mount attempt:
>
> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time
> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled
> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents
> [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5

Extent tree corruption.

So if that's the only corruption, you have a very high chance to recover most of your data.

Btrfs rescue can work, or you can try the experimental patches which provides rescue=skip_bg mount option to allow you mount the fs RO and receive your data (later is way faster than user space rescue)
https://patchwork.kernel.org/project/linux-btrfs/list/?series=130637

Also, for your dump super output, it doesn't provide too much info.

You would like to use -Ffa option for more info.
Also, you could also try that on all 3 devices, to find out which one has lower generation.

Also, please provide the history of the corruption. One generation corruptions is a little rare. Is sudden power loss involved in this case?

Thanks,
Qu

> [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed
>
> Output from 'btrfs check -p /dev/sdc1':
>
> # btrfs check -p /dev/sdc1
> Opening filesystem to check...
> parent transid verify failed on 229846466560 wanted 49749 found 49750
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2 > ERROR: cannot open file system
>
>
>
> On 08/15/2019 10:35 +0800, Qu Wenruo wrote:
>>>
>>>
>>> On 2019/8/15 ??????2:32, Tim Walberg wrote:
>>> > Most of the recommendations I've found online deal with when "wanted" is
>>> > greater than "found", which, if I understand correctly means that one or
>>> > more transactions were interrupted/lost before fully committed.
>>>
>>> No matter what the case is, a proper transaction shouldn't have any tree
>>> block overwritten.
>>>
>>> That means, either the FLUSH/FUA of the hardware/lower block layer is
>>> screwed up, or the COW of tree block is already screwed up.
>>>
>>> >
>>> > Are the recommendations for recovery the same if the system is reporting a
>>> > "wanted" that is less than "found"?
>>> >
>>> The salvage is no difference than any transid mismatch, no matter if
>>> it's larger or smaller.
>>>
>>> It depends on the tree block.
>>>
>>> Please provide full dmesg output and btrfs check for further advice.
>>>

Quoted 8 lines
Quoted 8 lines
	>>> 	Thanks,
	>>> 	Qu
	>>> 	
	> 
	> 
	> 
	> 

End of included message

Signature · 6 lines
+----------------------+
| Tim Walberg          |
| 830 Carriage Dr.     |
| Algonquin, IL 60102  |
| [email protected] |
+----------------------+

superblock: bytenr=65536, device=/dev/sdc1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x4331039b [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid 7338a973-9a45-4032-a4c9-d18142fd7908
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1407675531264
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3

superblock: bytenr=67108864, device=/dev/sdc1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xe3502b55 [match]
bytenr 67108864
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid 7338a973-9a45-4032-a4c9-d18142fd7908
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1407675531264
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3

superblock: bytenr=274877906944, device=/dev/sdc1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x1ed77d64 [match]
bytenr 274877906944
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid 7338a973-9a45-4032-a4c9-d18142fd7908
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1407675531264
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3

superblock: bytenr=65536, device=/dev/sdd1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xd0f74999 [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1404487860224
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 2
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3

superblock: bytenr=67108864, device=/dev/sdd1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x70966157 [match]
bytenr 67108864
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1404487860224
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 2
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3

superblock: bytenr=274877906944, device=/dev/sdd1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x8d113766 [match]
bytenr 274877906944
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1404487860224
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 2
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3

superblock: bytenr=65536, device=/dev/sde1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xb89a7a2e [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid ee71c327-ff63-4f46-8177-6328976f891f
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1405561602048
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 3
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3

superblock: bytenr=67108864, device=/dev/sde1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x18fb52e0 [match]
bytenr 67108864
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid ee71c327-ff63-4f46-8177-6328976f891f
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1405561602048
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 3
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3

superblock: bytenr=274877906944, device=/dev/sde1
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xe57c04d1 [match]
bytenr 274877906944
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 53749823-faaf-46f9-866d-3778d93cb1ca
label btrfs1
generation 49750
root 229846646784
sys_array_size 129
chunk_root_generation 49725
root_level 1
chunk_root 2568857059328
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 9001775738880
bytes_used 2085801975808
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 3
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x361
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 49748
uuid_tree_generation 49748
dev_item.uuid ee71c327-ff63-4f46-8177-6328976f891f
dev_item.fsid 53749823-faaf-46f9-866d-3778d93cb1ca [match]
dev_item.type 0
dev_item.total_bytes 3000591912960
dev_item.bytes_used 1405561602048
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 3
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:

	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 2568857059328)
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 2 offset 300690702336
			dev_uuid acad0cd5-aaec-4f6d-bf61-6c5b26888562
			stripe 1 devid 3 offset 1083179008
			dev_uuid ee71c327-ff63-4f46-8177-6328976f891f
backup_roots[4]:
	backup 0:
		backup_tree_root:	205750272	gen: 49747	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	174325760	gen: 49747	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	270893056	gen: 49747	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 1:
		backup_tree_root:	230054100992	gen: 49748	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	230054117376	gen: 49748	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	229931761664	gen: 49747	level: 1
		backup_csum_root:	230736379904	gen: 49748	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085801975808
		backup_num_devices:	3

	backup 2:
		backup_tree_root:	2031969501184	gen: 49745	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031960915968	gen: 49745	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2031960244224	gen: 49745	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802795008
		backup_num_devices:	3

	backup 3:
		backup_tree_root:	2032017784832	gen: 49746	level: 1
		backup_chunk_root:	2568857059328	gen: 49725	level: 1
		backup_extent_root:	2031988506624	gen: 49746	level: 2
		backup_fs_root:		850472665088	gen: 49727	level: 0
		backup_dev_root:	230821036032	gen: 49738	level: 1
		backup_csum_root:	2032023994368	gen: 49746	level: 3
		backup_total_bytes:	9001775738880
		backup_bytes_used:	2085802778624
		backup_num_devices:	3
#5 lore raw
Quoted 5 lines · Tim Walberg

'dump-super -Ffa' from all three devices attached.

'btrfs restore' did appear to recover most of the main data, minus snapshots, which would have greatly increased the required time and capacity, since I was recovering to XFS.

That's why I recommend that experimental patchset, it will make the fs mountable (RO though), with all btrfs snapshots available.

Quoted 3 lines

'btrfs rescue chunk-recover' ran, but failed to fix anything. 'btrfs rescue super-recover' says all supers are fine.

Those are useless for your case.

Quoted 4 lines

Initial corruption was due to a hard hang, which didn't leave enough crumbs to determine the source - might have been btrfs, might have been nvidia, might have been something completely different.

Anyway, the corruption is a little strange.

First of all, even hard hang/power loss shouldn't cause btrfs to overwrite its tree block, thus even hard hang/power loss happens, btrfs should be corrupted.

But that's definitely not the case. (We have quite some such report, but haven't pinned down the cause yet)

Secondly, the generation of your fs is strange.
The latest geneartion of your tree root is 49750, matches with your corrupted tree block, but your extent tree is definitely older.

So it looks like, your super blocks (all nine!) reach disk before some tree blocks reach the disk.

Finally, the superblock doesn't record previous transaction correctly. It doesn't has transaction of 49749 in its backup roots.

Not 100% sure, but looks somewhat like the problem fixed by this patch:
Btrfs: fix race leading to fs corruption after transaction abortion

It should get backported to all stable release recently.

Thanks,
Qu

Quoted 106 lines · Qu Wenruo
Quoted 88 lines
Quoted 88 lines · Tim Walberg

> Had to wait for 'btrfs recover' to finish before I proceed farther.
>
> Kernel is 4.19.45, tools are 4.19.1
>
> File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB)
>
> Output from attempting to mount:
>
> # mount -o ro,usebackuproot /dev/sdc1 /mnt
> mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
> missing codepage or helper program, or other error
>
> In some cases useful info is found in syslog - try
> dmesg | tail or so.
>
> Kernel messages from the mount attempt:
>
> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time
> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled
> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents
> [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750 > [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5

Extent tree corruption.

So if that's the only corruption, you have a very high chance to recover most of your data.

Btrfs rescue can work, or you can try the experimental patches which provides rescue=skip_bg mount option to allow you mount the fs RO and receive your data (later is way faster than user space rescue)
https://patchwork.kernel.org/project/linux-btrfs/list/?series=130637

Also, for your dump super output, it doesn't provide too much info.

You would like to use -Ffa option for more info.
Also, you could also try that on all 3 devices, to find out which one has lower generation.

Also, please provide the history of the corruption. One generation corruptions is a little rare. Is sudden power loss involved in this case?

Thanks,
Qu

> [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed
>
> Output from 'btrfs check -p /dev/sdc1':
>
> # btrfs check -p /dev/sdc1
> Opening filesystem to check...
> parent transid verify failed on 229846466560 wanted 49749 found 49750
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2 > ERROR: cannot open file system
>
>
>
> On 08/15/2019 10:35 +0800, Qu Wenruo wrote:
>>>
>>>
>>> On 2019/8/15 ??????2:32, Tim Walberg wrote:
>>> > Most of the recommendations I've found online deal with when "wanted" is
>>> > greater than "found", which, if I understand correctly means that one or
>>> > more transactions were interrupted/lost before fully committed.
>>>
>>> No matter what the case is, a proper transaction shouldn't have any tree
>>> block overwritten.
>>>
>>> That means, either the FLUSH/FUA of the hardware/lower block layer is
>>> screwed up, or the COW of tree block is already screwed up.
>>>
>>> >
>>> > Are the recommendations for recovery the same if the system is reporting a
>>> > "wanted" that is less than "found"?
>>> >
>>> The salvage is no difference than any transid mismatch, no matter if
>>> it's larger or smaller.
>>>
>>> It depends on the tree block.
>>>
>>> Please provide full dmesg output and btrfs check for further advice.
>>>

Quoted 8 lines
Quoted 8 lines
	>>> 	Thanks,
	>>> 	Qu
	>>> 	
	> 
	> 
	> 
	> 

End of included message

#6 lore raw

Thanks for all the help!

If I get a chance later today, I may try the patch set, but in the interest of getting things back online quicker, I may just have to recreate and restore the recovered data. The snapshots are no great loss - they're just one level of daily backups.

tw

Quoted 149 lines · Qu Wenruo
Quoted 149 lines · Tim Walberg
	> 'dump-super -Ffa' from all three devices attached.
	> 
	> 'btrfs restore' did appear to recover most of the main data, minus
	> snapshots, which would have greatly increased the required time and
	> capacity, since I was recovering to XFS.
	
	That's why I recommend that experimental patchset, it will make the fs
	mountable (RO though), with all btrfs snapshots available.
	
	> 
	> 'btrfs rescue chunk-recover' ran, but failed to fix anything.
	> 'btrfs rescue super-recover' says all supers are fine.
	
	Those are useless for your case.
	
	> 
	> Initial corruption was due to a hard hang, which didn't leave enough
	> crumbs to determine the source - might have been btrfs, might have
	> been nvidia, might have been something completely different.
	
	Anyway, the corruption is a little strange.
	
	First of all, even hard hang/power loss shouldn't cause btrfs to
	overwrite its tree block, thus even hard hang/power loss happens, btrfs
	should be corrupted.
	
	But that's definitely not the case. (We have quite some such report, but
	haven't pinned down the cause yet)
	
	Secondly, the generation of your fs is strange.
	The latest geneartion of your tree root is 49750, matches with your
	corrupted tree block, but your extent tree is definitely older.
	
	So it looks like, your super blocks (all nine!) reach disk before some
	tree blocks reach the disk.
	
	Finally, the superblock doesn't record previous transaction correctly.
	It doesn't has transaction of 49749 in its backup roots.
	
	Not 100% sure, but looks somewhat like the problem fixed by this patch:
	Btrfs: fix race leading to fs corruption after transaction abortion
	
	It should get backported to all stable release recently.
	
	Thanks,
	Qu
	
	> 
	> 
	> On 08/15/2019 22:07 +0800, Qu Wenruo wrote:
	>>> 	
	>>> 	
	>>> 	On 2019/8/15 ??????9:52, Tim Walberg wrote:
	>>> 	> Had to wait for 'btrfs recover' to finish before I proceed farther.
	>>> 	> 
	>>> 	> Kernel is 4.19.45, tools are 4.19.1
	>>> 	> 
	>>> 	> File system is a 3-disk RAID10 with WD3003FZEX (WD Black 3TB)
	>>> 	> 
	>>> 	> Output from attempting to mount:
	>>> 	> 
	>>> 	> # mount -o ro,usebackuproot /dev/sdc1 /mnt
	>>> 	> mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
	>>> 	>        missing codepage or helper program, or other error
	>>> 	> 
	>>> 	>        In some cases useful info is found in syslog - try
	>>> 	>        dmesg | tail or so.
	>>> 	> 
	>>> 	> Kernel messages from the mount attempt:
	>>> 	> 
	>>> 	> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): trying to use backup root at mount time
	>>> 	> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): disk space caching is enabled
	>>> 	> [Thu Aug 15 08:47:42 2019] BTRFS info (device sdc1): has skinny extents
	>>> 	> [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750
	>>> 	> [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): parent transid verify failed on 229846466560 wanted 49749 found 49750
	>>> 	> [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): failed to read block groups: -5
	>>> 	
	>>> 	Extent tree corruption.
	>>> 	
	>>> 	So if that's the only corruption, you have a very high chance to recover
	>>> 	most of your data.
	>>> 	
	>>> 	Btrfs rescue can work, or you can try the experimental patches which
	>>> 	provides rescue=skip_bg mount option to allow you mount the fs RO and
	>>> 	receive your data (later is way faster than user space rescue)
	>>> 	https://patchwork.kernel.org/project/linux-btrfs/list/?series=130637
	>>> 	
	>>> 	Also, for your dump super output, it doesn't provide too much info.
	>>> 	
	>>> 	You would like to use -Ffa option for more info.
	>>> 	Also, you could also try that on all 3 devices, to find out which one
	>>> 	has lower generation.
	>>> 	
	>>> 	Also, please provide the history of the corruption.
	>>> 	One generation corruptions is a little rare. Is sudden power loss
	>>> 	involved in this case?
	>>> 	
	>>> 	Thanks,
	>>> 	Qu
	>>> 	
	>>> 	> [Thu Aug 15 08:47:42 2019] BTRFS error (device sdc1): open_ctree failed
	>>> 	> 
	>>> 	> Output from 'btrfs check -p /dev/sdc1':
	>>> 	> 
	>>> 	> # btrfs check -p /dev/sdc1
	>>> 	> Opening filesystem to check...
	>>> 	> parent transid verify failed on 229846466560 wanted 49749 found 49750
	>>> 	> Ignoring transid failure
	>>> 	> ERROR: child eb corrupted: parent bytenr=229845336064 item=0 parent level=1 child level=2
	>>> 	> ERROR: cannot open file system
	>>> 	> 
	>>> 	> 
	>>> 	> 
	>>> 	> On 08/15/2019 10:35 +0800, Qu Wenruo wrote:
	>>> 	>>> 	
	>>> 	>>> 	
	>>> 	>>> 	On 2019/8/15 ??????2:32, Tim Walberg wrote:
	>>> 	>>> 	> Most of the recommendations I've found online deal with when "wanted" is
	>>> 	>>> 	> greater than "found", which, if I understand correctly means that one or
	>>> 	>>> 	> more transactions were interrupted/lost before fully committed.
	>>> 	>>> 	
	>>> 	>>> 	No matter what the case is, a proper transaction shouldn't have any tree
	>>> 	>>> 	block overwritten.
	>>> 	>>> 	
	>>> 	>>> 	That means, either the FLUSH/FUA of the hardware/lower block layer is
	>>> 	>>> 	screwed up, or the COW of tree block is already screwed up.
	>>> 	>>> 	
	>>> 	>>> 	> 
	>>> 	>>> 	> Are the recommendations for recovery the same if the system is reporting a
	>>> 	>>> 	> "wanted" that is less than "found"?
	>>> 	>>> 	> 
	>>> 	>>> 	The salvage is no difference than any transid mismatch, no matter if
	>>> 	>>> 	it's larger or smaller.
	>>> 	>>> 	
	>>> 	>>> 	It depends on the tree block.
	>>> 	>>> 	
	>>> 	>>> 	Please provide full dmesg output and btrfs check for further advice.
	>>> 	>>> 	
	>>> 	>>> 	Thanks,
	>>> 	>>> 	Qu
	>>> 	>>> 	
	>>> 	> 
	>>> 	> 
	>>> 	> 
	>>> 	> 
	>>> 	
Quoted 8 lines
Quoted 8 lines
	> 
	> 
	> 
	> End of included message
	> 
	> 
	> 

End of included message

Signature · 6 lines
+----------------------+
| Tim Walberg          |
| 830 Carriage Dr.     |
| Algonquin, IL 60102  |
| [email protected] |
+----------------------+