Skip to contentSkip to conversation

Null deref during attempted replay of corrupt TREE_LOG in newer kernel

DISCUSSION Russell Haley 3 participants 6 messages Jul 13, 2025 – Jul 24, 2025 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Howdy,

After a power outage, my computer booted into emergency mode, apparently due to a corrupt log tree on the root SSD (WD Blue, WDC WDS500G2B0C-00PXH0). Symptoms basically identical to [1], and I was eventually able to fix it the same way, with btrfs rescue zero-log.

*However*, troubleshooting was complicated by the fact that, on kernel 6.15.4, attempting to mount the corrupt FS even once OOPSed the kernel and locked the device, making it impossible to run btrfs commands against it or try different mount options. This would've been a showstopper if I didn't have Fedora's backup kernels to try and see that the dmesg error was different. That led me to boot into a live USB with kernel 6.14.0, from which I was able to investigate. I then booted into backup kernel 6.14.9 (the newest non-problematic) and zeroed the log.

Facts:

1. Kernel 6.15.4 has the null deref.

2. Kernel 6.14.9 does not.

3. mount -o ro,rescue=nologreplay succeeds on kernel 6.14.0 (from live USB) and 6.15.4 (newer live USB). On 6.15.4, it succeeds if it is the first mount command run.

Here's all the evidence I collected before zeroing the log so I could have a working PC again. It's the log from the bad kernel, one good kernel, and btrfs inspect-internal dump-super and dump-tree of the corrupted disk

Log from kernel 6.15.4:

Quoted 81 lines
Jul 13 04:25:26 localhost-live kernel: BTRFS: device label fedora_hogwarts devid 1 transid 456817 /dev/mapper/hogwarts-root (252:1) scanned by mount (4794)
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): first mount of filesystem a6999502-9362-4fa7-b7b4-28095790217d
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): using crc32c (crc32c-x86_64) checksum algorithm
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): using free-space-tree
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): bdev /dev/mapper/hogwarts-root errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): start tree-log replay
Jul 13 04:25:26 localhost-live kernel: BUG: kernel NULL pointer dereference, address: 0000000000000219
Jul 13 04:25:26 localhost-live kernel: #PF: supervisor read access in kernel mode
Jul 13 04:25:26 localhost-live kernel: #PF: error_code(0x0000) - not-present page
Jul 13 04:25:26 localhost-live kernel: PGD 0 P4D 0
Jul 13 04:25:26 localhost-live kernel: Oops: Oops: 0000 [#1] SMP PTI
Jul 13 04:25:26 localhost-live kernel: CPU: 0 UID: 0 PID: 4794 Comm: mount Not tainted 6.15.4-200.fc42.x86_64 #1 PREEMPT(lazy)
Jul 13 04:25:26 localhost-live kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme4, BIOS P3.50 03/11/2018
Jul 13 04:25:26 localhost-live kernel: RIP: 0010:iput+0x20/0x230
Jul 13 04:25:26 localhost-live kernel: Code: 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 86 01 00 00 41 54 55 48 8d af 50 01 00 00 53 48 89 fb <f6> 87 91 00 00 00 01 74 3a e9 68 01 00 00 8b 53 48 8b 83 90 00 00
Jul 13 04:25:26 localhost-live kernel: RSP: 0018:ffffd43c0b893740 EFLAGS: 00010206
Jul 13 04:25:26 localhost-live kernel: RAX: 0000000000000000 RBX: 0000000000000188 RCX: 000000000092a000
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: ffffffff93bcea60 RDI: 0000000000000188
Jul 13 04:25:26 localhost-live kernel: RBP: 00000000000002d8 R08: 0000000000000000 R09: ffffffff90db25b6
Jul 13 04:25:26 localhost-live kernel: R10: ffff8c4aa9480000 R11: fffff60205a52000 R12: ffffd43c0b8938bf
Jul 13 04:25:26 localhost-live kernel: R13: 0000000000000000 R14: ffff8c4aa9476690 R15: 00000000fffffffb
Jul 13 04:25:26 localhost-live kernel: FS:  00007f1b5f3f9840(0000) GS:ffff8c4fc3a71000(0000) knlGS:0000000000000000
Jul 13 04:25:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219 CR3: 000000016abc8003 CR4: 00000000001726f0
Jul 13 04:25:26 localhost-live kernel: Call Trace:
Jul 13 04:25:26 localhost-live kernel:  <TASK>
Jul 13 04:25:26 localhost-live kernel:  replay_one_extent+0xba/0x740
Jul 13 04:25:26 localhost-live kernel:  ? release_extent_buffer+0xb5/0xd0
Jul 13 04:25:26 localhost-live kernel:  ? copy_extent_buffer+0x126/0x160
Jul 13 04:25:26 localhost-live kernel:  ? btrfs_release_path+0x2b/0x1b0
Jul 13 04:25:26 localhost-live kernel:  replay_one_buffer+0x2ee/0x500
Jul 13 04:25:26 localhost-live kernel:  ? alloc_extent_buffer+0x97/0x500
Jul 13 04:25:26 localhost-live kernel:  ? read_extent_buffer+0x102/0x140
Jul 13 04:25:26 localhost-live kernel:  walk_down_log_tree+0x1d8/0x4c0
Jul 13 04:25:26 localhost-live kernel:  ? kmem_cache_alloc_noprof+0x141/0x410
Jul 13 04:25:26 localhost-live kernel:  walk_log_tree+0xe6/0x280
Jul 13 04:25:26 localhost-live kernel:  btrfs_recover_log_trees+0x1cc/0x5a0
Jul 13 04:25:26 localhost-live kernel:  ? __pfx_replay_one_buffer+0x10/0x10
Jul 13 04:25:26 localhost-live kernel:  open_ctree+0x912/0xb98
Jul 13 04:25:26 localhost-live kernel:  btrfs_get_tree_super.cold+0xb/0xbf
Jul 13 04:25:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:25:26 localhost-live kernel:  fc_mount+0x12/0x40
Jul 13 04:25:26 localhost-live kernel:  btrfs_get_tree_subvol+0x10d/0x210
Jul 13 04:25:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:25:26 localhost-live kernel:  vfs_cmd_create+0x57/0xd0
Jul 13 04:25:26 localhost-live kernel:  __do_sys_fsconfig+0x4b1/0x640
Jul 13 04:25:26 localhost-live kernel:  do_syscall_64+0x7b/0x160
Jul 13 04:25:26 localhost-live kernel:  ? __check_object_size.part.0+0x59/0xc0
Jul 13 04:25:26 localhost-live kernel:  ? _copy_from_user+0x27/0x60
Jul 13 04:25:26 localhost-live kernel:  ? lookup_constant+0x35/0x50
Jul 13 04:25:26 localhost-live kernel:  ? vfs_parse_fs_param+0x30/0x100
Jul 13 04:25:26 localhost-live kernel:  ? __do_sys_fsconfig+0x354/0x640
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? __do_sys_newfstatat+0x4b/0x80
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? exc_page_fault+0x7e/0x1a0
Jul 13 04:25:26 localhost-live kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 13 04:25:26 localhost-live kernel: RIP: 0033:0x7f1b5f5d7abe
Jul 13 04:25:26 localhost-live kernel: Code: 73 01 c3 48 8b 0d 4a 33 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1a 33 0f 00 f7 d8 64 89 01 48
Jul 13 04:25:26 localhost-live kernel: RSP: 002b:00007ffe53604898 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
Jul 13 04:25:26 localhost-live kernel: RAX: ffffffffffffffda RBX: 0000556b3a8eb560 RCX: 00007f1b5f5d7abe
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
Jul 13 04:25:26 localhost-live kernel: RBP: 00007ffe536049e0 R08: 0000000000000000 R09: 0000000000000003
Jul 13 04:25:26 localhost-live kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Jul 13 04:25:26 localhost-live kernel: R13: 0000556b3a8eba00 R14: 00007f1b5f754b00 R15: 0000556b3a8ed498
Jul 13 04:25:26 localhost-live kernel:  </TASK>
Jul 13 04:25:26 localhost-live kernel: Modules linked in: f2fs dm_crypt uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi bnep snd_hda_scodec_component snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi btusb snd_hda_codec btrtl btintel intel_rapl_msr snd_hda_core btbcm uvcvideo intel_rapl_common snd_usb_audio btmtk snd_usbmidi_lib snd_ump snd_rawmidi x86_pkg_temp_thermal snd_hwdep intel_powerclamp bluetooth snd_seq uvc videobuf2_vmalloc coretemp videobuf2_memops videobuf2_v4l2 kvm_intel snd_seq_device videobuf2_common snd_pcm videodev mei_hdcp mei_pxp iTCO_wdt intel_pmc_bxt at24 snd_timer mei_me iTCO_vendor_support kvm snd mei mc i2c_i801 joydev i2c_smbus pcspkr lpc_ich rfkill apple_mfi_fastcharge irqbypass rapl intel_cstate intel_uncore soundcore nfnetlink
Jul 13 04:25:26 localhost-live kernel:  zram lz4hc_compress lz4_compress overlay erofs netfs hid_logitech_hidpp isofs amdgpu hid_logitech_dj i915 mxm_wmi amdxcp polyval_clmulni gpu_sched polyval_generic drm_panel_backlight_quirks ghash_clmulni_intel drm_ttm_helper drm_buddy sha512_ssse3 ttm drm_exec sha256_ssse3 i2c_algo_bit uas drm_suballoc_helper sha1_ssse3 hid_apple usb_storage drm_display_helper e1000e nvme cec nvme_tcp nvme_fabrics nvme_core video wmi nvme_keyring nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219
Jul 13 04:25:26 localhost-live kernel: ---[ end trace 0000000000000000 ]---
Jul 13 04:25:26 localhost-live kernel: RIP: 0010:iput+0x20/0x230
Jul 13 04:25:26 localhost-live kernel: Code: 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 86 01 00 00 41 54 55 48 8d af 50 01 00 00 53 48 89 fb <f6> 87 91 00 00 00 01 74 3a e9 68 01 00 00 8b 53 48 8b 83 90 00 00
Jul 13 04:25:26 localhost-live kernel: RSP: 0018:ffffd43c0b893740 EFLAGS: 00010206
Quoted 8 lines
Jul 13 04:25:26 localhost-live kernel: RAX: 0000000000000000 RBX: 0000000000000188 RCX: 000000000092a000
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: ffffffff93bcea60 RDI: 0000000000000188
Jul 13 04:25:26 localhost-live kernel: RBP: 00000000000002d8 R08: 0000000000000000 R09: ffffffff90db25b6
Jul 13 04:25:26 localhost-live kernel: R10: ffff8c4aa9480000 R11: fffff60205a52000 R12: ffffd43c0b8938bf
Jul 13 04:25:26 localhost-live kernel: R13: 0000000000000000 R14: ffff8c4aa9476690 R15: 00000000fffffffb
Jul 13 04:25:26 localhost-live kernel: FS:  00007f1b5f3f9840(0000) GS:ffff8c4fc3a71000(0000) knlGS:0000000000000000
Jul 13 04:25:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219 CR3: 000000016abc8003 CR4: 00000000001726f0

Log from kernel 6.14.0:

Quoted 60 lines
Jul 13 04:02:25 localhost-live kernel: BTRFS: device label fedora_hogwarts devid 1 transid 456817 /dev/mapper/hogwarts-root (252:1) scanned by mount (4785)
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): first mount of filesystem a6999502-9362-4fa7-b7b4-28095790217d
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): using crc32c (crc32c-x86_64) checksum algorithm
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): using free-space-tree
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): bdev /dev/mapper/hogwarts-root errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): start tree-log replay
Jul 13 04:02:26 localhost-live kernel: BTRFS: error (device dm-1) in btrfs_replay_log:2104: errno=-5 IO failure (Failed to recover log tree)
Jul 13 04:02:26 localhost-live kernel: ------------[ cut here ]------------
Jul 13 04:02:26 localhost-live kernel: WARNING: CPU: 0 PID: 4785 at fs/btrfs/block-rsv.c:452 btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel: Modules linked in: dm_crypt uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp qrtr coretemp kvm_intel snd_hda_codec_realtek snd_hda_codec_generic bnep snd_hda_codec_hdmi kvm snd_usb_audio btusb snd_hda_scodec_component snd_hda_intel btrtl btintel snd_intel_dspcfg snd_intel_sdw_acpi btbcm snd_hda_codec btmtk bluetooth uvcvideo snd_hda_core snd_usbmidi_lib snd_ump snd_rawmidi snd_hwdep snd_seq uvc videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_seq_device videobuf2_common snd_pcm videodev mei_hdcp mei_pxp iTCO_wdt at24 snd_timer mei_me intel_pmc_bxt iTCO_vendor_support mei snd rapl intel_cstate intel_uncore mc i2c_i801 lpc_ich rfkill i2c_smbus pcspkr soundcore apple_mfi_fastcharge joydev nfnetlink zram
Jul 13 04:02:26 localhost-live kernel:  lz4hc_compress lz4_compress overlay erofs netfs isofs hid_logitech_hidpp uas hid_logitech_dj usb_storage hid_apple amdgpu i915 amdxcp gpu_sched drm_panel_backlight_quirks drm_buddy drm_ttm_helper mxm_wmi ttm polyval_clmulni drm_exec polyval_generic ghash_clmulni_intel i2c_algo_bit drm_suballoc_helper sha512_ssse3 sha256_ssse3 drm_display_helper sha1_ssse3 nvme e1000e nvme_tcp cec nvme_fabrics video nvme_keyring wmi nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev
Jul 13 04:02:26 localhost-live kernel: CPU: 0 UID: 0 PID: 4785 Comm: mount Not tainted 6.14.0-63.fc42.x86_64 #1
Jul 13 04:02:26 localhost-live kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme4, BIOS P3.50 03/11/2018
Jul 13 04:02:26 localhost-live kernel: RIP: 0010:btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel: Code: 01 00 00 00 74 aa 0f 0b 48 83 bb 40 01 00 00 00 74 a8 0f 0b 48 83 bb 48 01 00 00 00 74 a6 0f 0b 48 83 bb 70 01 00 00 00 74 a4 <0f> 0b 48 83 bb 78 01 00 00 00 74 a2 0f 0b 48 83 bb a8 01 00 00 00
Jul 13 04:02:26 localhost-live kernel: RSP: 0018:ffffb1918a49bc38 EFLAGS: 00010286
Jul 13 04:02:26 localhost-live kernel: RAX: 0000000020000000 RBX: ffff9271822e3000 RCX: 0000000000000000
Jul 13 04:02:26 localhost-live kernel: RDX: 0000000020000000 RSI: ffff92723e622c00 RDI: ffff92723e622c08
Jul 13 04:02:26 localhost-live kernel: RBP: ffff927555127000 R08: ffff927555127000 R09: 0000000000100000
Jul 13 04:02:26 localhost-live kernel: R10: ffff9271c30575c8 R11: 0000000000000036 R12: ffff9271822e3090
Jul 13 04:02:26 localhost-live kernel: R13: ffff9271822e36c8 R14: 0000000000000000 R15: dead000000000100
Jul 13 04:02:26 localhost-live kernel: FS:  00007f67bbcda840(0000) GS:ffff927697600000(0000) knlGS:0000000000000000
Jul 13 04:02:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:02:26 localhost-live kernel: CR2: 00007fc484069000 CR3: 0000000175292002 CR4: 00000000001726f0
Jul 13 04:02:26 localhost-live kernel: Call Trace:
Jul 13 04:02:26 localhost-live kernel:  <TASK>
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_free_block_groups+0x34a/0x420
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? __warn.cold+0x93/0xfa
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? report_bug+0x103/0x150
Jul 13 04:02:26 localhost-live kernel:  ? handle_bug+0x58/0x90
Jul 13 04:02:26 localhost-live kernel:  ? exc_invalid_op+0x17/0x70
Jul 13 04:02:26 localhost-live kernel:  ? asm_exc_invalid_op+0x1a/0x20
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0x22/0xd0
Jul 13 04:02:26 localhost-live kernel:  btrfs_free_block_groups+0x34a/0x420
Jul 13 04:02:26 localhost-live kernel:  open_ctree+0xb40/0xbce
Jul 13 04:02:26 localhost-live kernel:  btrfs_get_tree_super.cold+0xb/0xbb
Jul 13 04:02:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:02:26 localhost-live kernel:  fc_mount+0x12/0x40
Jul 13 04:02:26 localhost-live kernel:  btrfs_get_tree_subvol+0x10d/0x210
Jul 13 04:02:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:02:26 localhost-live kernel:  vfs_cmd_create+0x59/0xe0
Jul 13 04:02:26 localhost-live kernel:  __do_sys_fsconfig+0x511/0x6f0
Jul 13 04:02:26 localhost-live kernel:  do_syscall_64+0x7f/0x170
Jul 13 04:02:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? do_readlinkat+0x12e/0x180
Jul 13 04:02:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? exc_page_fault+0x7e/0x1a0
Jul 13 04:02:26 localhost-live kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 13 04:02:26 localhost-live kernel: RIP: 0033:0x7f67bbeb9abe
Jul 13 04:02:26 localhost-live kernel: Code: 73 01 c3 48 8b 0d 4a 33 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1a 33 0f 00 f7 d8 64 89 01 48
Jul 13 04:02:26 localhost-live kernel: RSP: 002b:00007ffd51c98aa8 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
Quoted 9 lines
Jul 13 04:02:26 localhost-live kernel: RAX: ffffffffffffffda RBX: 000055ecc1aaa560 RCX: 00007f67bbeb9abe
Jul 13 04:02:26 localhost-live kernel: RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
Jul 13 04:02:26 localhost-live kernel: RBP: 00007ffd51c98bf0 R08: 0000000000000000 R09: 0000000000000003
Jul 13 04:02:26 localhost-live kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Jul 13 04:02:26 localhost-live kernel: R13: 000055ecc1aaaa00 R14: 00007f67bc036b00 R15: 000055ecc1aac498
Jul 13 04:02:26 localhost-live kernel:  </TASK>
Jul 13 04:02:26 localhost-live kernel: ---[ end trace 0000000000000000 ]---
Jul 13 04:02:26 localhost-live kernel: BTRFS error (device dm-1 state E): open_ctree failed: -5

Superblock dump collected with btrfs-tools 6.15 on live USB:

Quoted 99 lines

btrfs inspect-internal dump-super --full /dev/mapper/hogwarts-root
superblock: bytenr=65536, device=/dev/mapper/hogwarts-root
---------------------------------------------------------

csum_type               0 (crc32c)
csum_size               4
csum                    0x3e3b7bc2 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    a6999502-9362-4fa7-b7b4-28095790217d
metadata_uuid           00000000-0000-0000-0000-000000000000
label                   fedora_hogwarts
generation              456817
root                    202624811008
sys_array_size          97
chunk_root_generation   433110
root_level              0
chunk_root              368550690816
chunk_root_level        1
log_root                202650910720
log_root_transid (deprecated)   0
log_root_level          0
total_bytes             498382929920
bytes_used              227109515264
sectorsize              4096
nodesize                16384
leafsize (deprecated)   16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x3

( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID )
incompat_flags 0x171
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )

cache_generation        0
uuid_tree_generation    456817
dev_item.uuid           0721004b-8c3e-4ae7-bb3b-4b2bb5d313f7
dev_item.fsid           a6999502-9362-4fa7-b7b4-28095790217d [match]
dev_item.type           0
dev_item.total_bytes    498382929920
dev_item.bytes_used     253453402112
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 368550346752)
                length 33554432 owner 2 stripe_len 65536 type SYSTEM|single
                io_align 65536 io_width 65536 sector_size 4096
                num_stripes 1 sub_stripes 1
                        stripe 0 devid 1 offset 79512469504
                        dev_uuid 0721004b-8c3e-4ae7-bb3b-4b2bb5d313f7
backup_roots[4]:
        backup 0:
                backup_tree_root:       202624811008    gen: 456817     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202621485056    gen: 456817     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202621796352    gen: 456817     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227109515264
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       202407755776    gen: 456814     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202402742272    gen: 456814     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202403151872    gen: 456814     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227106684928
                backup_num_devices:     1

        backup 2:
                backup_tree_root:       202472538112    gen: 456815     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202447388672    gen: 456815     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202378985472    gen: 456815     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227107676160
                backup_num_devices:     1

        backup 3:
                backup_tree_root:       202539106304    gen: 456816     level: 0
Quoted 8 lines
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202537730048    gen: 456816     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202537861120    gen: 456816     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227108405248
                backup_num_devices:     1

Corrupt tree log dump collected with btrfs-tools 6.15 on live USB:

Quoted 25 lines
# btrfs inspect-internal dump-tree -t TREE_LOG /dev/mapper/hogwarts-root
btrfs-progs v6.15
log root tree
leaf 202650910720 items 2 free space 15355 generation 456818 owner TREE_LOG
leaf 202650910720 flags 0x1(WRITTEN) backref revision 1
fs uuid a6999502-9362-4fa7-b7b4-28095790217d
chunk uuid 710679da-fb76-4a51-9c84-8600a6e58c43
        item 0 key (TREE_LOG ROOT_ITEM 256) itemoff 15844 itemsize 439
                generation 456818 root_dirid 0 bytenr 202645028864 byte_limit 0 bytes_used 0
                last_snapshot 0 flags 0x0(none) refs 0
                drop_progress key (0 UNKNOWN.0 0) drop_level 0
                level 0 generation_v2 456818
                uuid 00000000-0000-0000-0000-000000000000
                parent_uuid 00000000-0000-0000-0000-000000000000
                received_uuid 00000000-0000-0000-0000-000000000000
                ctransid 0 otransid 0 stransid 0 rtransid 0
                ctime 0.0 (1970-01-01 00:00:00)
                otime 0.0 (1970-01-01 00:00:00)
                stime 0.0 (1970-01-01 00:00:00)
                rtime 0.0 (1970-01-01 00:00:00)
        item 1 key (TREE_LOG ROOT_ITEM 258) itemoff 15405 itemsize 439
                generation 456818 root_dirid 0 bytenr 202650877952 byte_limit 0 bytes_used 65536
                last_snapshot 0 flags 0x0(none) refs 0
                drop_progress key (0 UNKNOWN.0 0) drop_level 0
                level 1 generation_v2 456818
Quoted 8 lines
                uuid 00000000-0000-0000-0000-000000000000
                parent_uuid 00000000-0000-0000-0000-000000000000
                received_uuid 00000000-0000-0000-0000-000000000000
                ctransid 0 otransid 0 stransid 0 rtransid 0
                ctime 0.0 (1970-01-01 00:00:00)
                otime 0.0 (1970-01-01 00:00:00)
                stime 0.0 (1970-01-01 00:00:00)
                rtime 0.0 (1970-01-01 00:00:00)

If I'm interpreting correctly, the corrupt tree log is zeroed, so that kind of gestures in the direction of what the log replay code might be mishandling.
[1]
https://discussion.fedoraproject.org/t/btrfs-error-emergency-mode-entered-every-time-i-boot/141179

Thanks,

Russell

#1 lore raw

Gentle ping.

@Filipe You've recently submitted a cleanup of the log replay code, so I assume you're familiar with it. Is there anything Funny Lookin' in the path that handles a corrupt TREE_LOG? Would've changed sometime between 6.14.9 and 6.15.4.

Thanks,

Russell

Quoted 350 lines · Russell Haley

Howdy,

After a power outage, my computer booted into emergency mode, apparently due to a corrupt log tree on the root SSD (WD Blue, WDC WDS500G2B0C-00PXH0). Symptoms basically identical to [1], and I was eventually able to fix it the same way, with btrfs rescue zero-log.

*However*, troubleshooting was complicated by the fact that, on kernel 6.15.4, attempting to mount the corrupt FS even once OOPSed the kernel and locked the device, making it impossible to run btrfs commands against it or try different mount options. This would've been a showstopper if I didn't have Fedora's backup kernels to try and see that the dmesg error was different. That led me to boot into a live USB with kernel 6.14.0, from which I was able to investigate. I then booted into backup kernel 6.14.9 (the newest non-problematic) and zeroed the log.

Facts:

1. Kernel 6.15.4 has the null deref.

2. Kernel 6.14.9 does not.

3. mount -o ro,rescue=nologreplay succeeds on kernel 6.14.0 (from live USB) and 6.15.4 (newer live USB). On 6.15.4, it succeeds if it is the first mount command run.

Here's all the evidence I collected before zeroing the log so I could have a working PC again. It's the log from the bad kernel, one good kernel, and btrfs inspect-internal dump-super and dump-tree of the corrupted disk

Log from kernel 6.15.4:

Quoted 81 lines
Jul 13 04:25:26 localhost-live kernel: BTRFS: device label fedora_hogwarts devid 1 transid 456817 /dev/mapper/hogwarts-root (252:1) scanned by mount (4794)
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): first mount of filesystem a6999502-9362-4fa7-b7b4-28095790217d
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): using crc32c (crc32c-x86_64) checksum algorithm
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): using free-space-tree
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): bdev /dev/mapper/hogwarts-root errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): start tree-log replay
Jul 13 04:25:26 localhost-live kernel: BUG: kernel NULL pointer dereference, address: 0000000000000219
Jul 13 04:25:26 localhost-live kernel: #PF: supervisor read access in kernel mode
Jul 13 04:25:26 localhost-live kernel: #PF: error_code(0x0000) - not-present page
Jul 13 04:25:26 localhost-live kernel: PGD 0 P4D 0
Jul 13 04:25:26 localhost-live kernel: Oops: Oops: 0000 [#1] SMP PTI
Jul 13 04:25:26 localhost-live kernel: CPU: 0 UID: 0 PID: 4794 Comm: mount Not tainted 6.15.4-200.fc42.x86_64 #1 PREEMPT(lazy)
Jul 13 04:25:26 localhost-live kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme4, BIOS P3.50 03/11/2018
Jul 13 04:25:26 localhost-live kernel: RIP: 0010:iput+0x20/0x230
Jul 13 04:25:26 localhost-live kernel: Code: 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 86 01 00 00 41 54 55 48 8d af 50 01 00 00 53 48 89 fb <f6> 87 91 00 00 00 01 74 3a e9 68 01 00 00 8b 53 48 8b 83 90 00 00
Jul 13 04:25:26 localhost-live kernel: RSP: 0018:ffffd43c0b893740 EFLAGS: 00010206
Jul 13 04:25:26 localhost-live kernel: RAX: 0000000000000000 RBX: 0000000000000188 RCX: 000000000092a000
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: ffffffff93bcea60 RDI: 0000000000000188
Jul 13 04:25:26 localhost-live kernel: RBP: 00000000000002d8 R08: 0000000000000000 R09: ffffffff90db25b6
Jul 13 04:25:26 localhost-live kernel: R10: ffff8c4aa9480000 R11: fffff60205a52000 R12: ffffd43c0b8938bf
Jul 13 04:25:26 localhost-live kernel: R13: 0000000000000000 R14: ffff8c4aa9476690 R15: 00000000fffffffb
Jul 13 04:25:26 localhost-live kernel: FS:  00007f1b5f3f9840(0000) GS:ffff8c4fc3a71000(0000) knlGS:0000000000000000
Jul 13 04:25:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219 CR3: 000000016abc8003 CR4: 00000000001726f0
Jul 13 04:25:26 localhost-live kernel: Call Trace:
Jul 13 04:25:26 localhost-live kernel:  <TASK>
Jul 13 04:25:26 localhost-live kernel:  replay_one_extent+0xba/0x740
Jul 13 04:25:26 localhost-live kernel:  ? release_extent_buffer+0xb5/0xd0
Jul 13 04:25:26 localhost-live kernel:  ? copy_extent_buffer+0x126/0x160
Jul 13 04:25:26 localhost-live kernel:  ? btrfs_release_path+0x2b/0x1b0
Jul 13 04:25:26 localhost-live kernel:  replay_one_buffer+0x2ee/0x500
Jul 13 04:25:26 localhost-live kernel:  ? alloc_extent_buffer+0x97/0x500
Jul 13 04:25:26 localhost-live kernel:  ? read_extent_buffer+0x102/0x140
Jul 13 04:25:26 localhost-live kernel:  walk_down_log_tree+0x1d8/0x4c0
Jul 13 04:25:26 localhost-live kernel:  ? kmem_cache_alloc_noprof+0x141/0x410
Jul 13 04:25:26 localhost-live kernel:  walk_log_tree+0xe6/0x280
Jul 13 04:25:26 localhost-live kernel:  btrfs_recover_log_trees+0x1cc/0x5a0
Jul 13 04:25:26 localhost-live kernel:  ? __pfx_replay_one_buffer+0x10/0x10
Jul 13 04:25:26 localhost-live kernel:  open_ctree+0x912/0xb98
Jul 13 04:25:26 localhost-live kernel:  btrfs_get_tree_super.cold+0xb/0xbf
Jul 13 04:25:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:25:26 localhost-live kernel:  fc_mount+0x12/0x40
Jul 13 04:25:26 localhost-live kernel:  btrfs_get_tree_subvol+0x10d/0x210
Jul 13 04:25:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:25:26 localhost-live kernel:  vfs_cmd_create+0x57/0xd0
Jul 13 04:25:26 localhost-live kernel:  __do_sys_fsconfig+0x4b1/0x640
Jul 13 04:25:26 localhost-live kernel:  do_syscall_64+0x7b/0x160
Jul 13 04:25:26 localhost-live kernel:  ? __check_object_size.part.0+0x59/0xc0
Jul 13 04:25:26 localhost-live kernel:  ? _copy_from_user+0x27/0x60
Jul 13 04:25:26 localhost-live kernel:  ? lookup_constant+0x35/0x50
Jul 13 04:25:26 localhost-live kernel:  ? vfs_parse_fs_param+0x30/0x100
Jul 13 04:25:26 localhost-live kernel:  ? __do_sys_fsconfig+0x354/0x640
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? __do_sys_newfstatat+0x4b/0x80
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? exc_page_fault+0x7e/0x1a0
Jul 13 04:25:26 localhost-live kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 13 04:25:26 localhost-live kernel: RIP: 0033:0x7f1b5f5d7abe
Jul 13 04:25:26 localhost-live kernel: Code: 73 01 c3 48 8b 0d 4a 33 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1a 33 0f 00 f7 d8 64 89 01 48
Jul 13 04:25:26 localhost-live kernel: RSP: 002b:00007ffe53604898 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
Jul 13 04:25:26 localhost-live kernel: RAX: ffffffffffffffda RBX: 0000556b3a8eb560 RCX: 00007f1b5f5d7abe
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
Jul 13 04:25:26 localhost-live kernel: RBP: 00007ffe536049e0 R08: 0000000000000000 R09: 0000000000000003
Jul 13 04:25:26 localhost-live kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Jul 13 04:25:26 localhost-live kernel: R13: 0000556b3a8eba00 R14: 00007f1b5f754b00 R15: 0000556b3a8ed498
Jul 13 04:25:26 localhost-live kernel:  </TASK>
Jul 13 04:25:26 localhost-live kernel: Modules linked in: f2fs dm_crypt uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi bnep snd_hda_scodec_component snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi btusb snd_hda_codec btrtl btintel intel_rapl_msr snd_hda_core btbcm uvcvideo intel_rapl_common snd_usb_audio btmtk snd_usbmidi_lib snd_ump snd_rawmidi x86_pkg_temp_thermal snd_hwdep intel_powerclamp bluetooth snd_seq uvc videobuf2_vmalloc coretemp videobuf2_memops videobuf2_v4l2 kvm_intel snd_seq_device videobuf2_common snd_pcm videodev mei_hdcp mei_pxp iTCO_wdt intel_pmc_bxt at24 snd_timer mei_me iTCO_vendor_support kvm snd mei mc i2c_i801 joydev i2c_smbus pcspkr lpc_ich rfkill apple_mfi_fastcharge irqbypass rapl intel_cstate intel_uncore soundcore nfnetlink
Jul 13 04:25:26 localhost-live kernel:  zram lz4hc_compress lz4_compress overlay erofs netfs hid_logitech_hidpp isofs amdgpu hid_logitech_dj i915 mxm_wmi amdxcp polyval_clmulni gpu_sched polyval_generic drm_panel_backlight_quirks ghash_clmulni_intel drm_ttm_helper drm_buddy sha512_ssse3 ttm drm_exec sha256_ssse3 i2c_algo_bit uas drm_suballoc_helper sha1_ssse3 hid_apple usb_storage drm_display_helper e1000e nvme cec nvme_tcp nvme_fabrics nvme_core video wmi nvme_keyring nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219
Jul 13 04:25:26 localhost-live kernel: ---[ end trace 0000000000000000 ]---
Jul 13 04:25:26 localhost-live kernel: RIP: 0010:iput+0x20/0x230
Jul 13 04:25:26 localhost-live kernel: Code: 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 86 01 00 00 41 54 55 48 8d af 50 01 00 00 53 48 89 fb <f6> 87 91 00 00 00 01 74 3a e9 68 01 00 00 8b 53 48 8b 83 90 00 00
Jul 13 04:25:26 localhost-live kernel: RSP: 0018:ffffd43c0b893740 EFLAGS: 00010206
Quoted 8 lines
Jul 13 04:25:26 localhost-live kernel: RAX: 0000000000000000 RBX: 0000000000000188 RCX: 000000000092a000
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: ffffffff93bcea60 RDI: 0000000000000188
Jul 13 04:25:26 localhost-live kernel: RBP: 00000000000002d8 R08: 0000000000000000 R09: ffffffff90db25b6
Jul 13 04:25:26 localhost-live kernel: R10: ffff8c4aa9480000 R11: fffff60205a52000 R12: ffffd43c0b8938bf
Jul 13 04:25:26 localhost-live kernel: R13: 0000000000000000 R14: ffff8c4aa9476690 R15: 00000000fffffffb
Jul 13 04:25:26 localhost-live kernel: FS:  00007f1b5f3f9840(0000) GS:ffff8c4fc3a71000(0000) knlGS:0000000000000000
Jul 13 04:25:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219 CR3: 000000016abc8003 CR4: 00000000001726f0

Log from kernel 6.14.0:

Quoted 60 lines
Jul 13 04:02:25 localhost-live kernel: BTRFS: device label fedora_hogwarts devid 1 transid 456817 /dev/mapper/hogwarts-root (252:1) scanned by mount (4785)
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): first mount of filesystem a6999502-9362-4fa7-b7b4-28095790217d
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): using crc32c (crc32c-x86_64) checksum algorithm
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): using free-space-tree
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): bdev /dev/mapper/hogwarts-root errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): start tree-log replay
Jul 13 04:02:26 localhost-live kernel: BTRFS: error (device dm-1) in btrfs_replay_log:2104: errno=-5 IO failure (Failed to recover log tree)
Jul 13 04:02:26 localhost-live kernel: ------------[ cut here ]------------
Jul 13 04:02:26 localhost-live kernel: WARNING: CPU: 0 PID: 4785 at fs/btrfs/block-rsv.c:452 btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel: Modules linked in: dm_crypt uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp qrtr coretemp kvm_intel snd_hda_codec_realtek snd_hda_codec_generic bnep snd_hda_codec_hdmi kvm snd_usb_audio btusb snd_hda_scodec_component snd_hda_intel btrtl btintel snd_intel_dspcfg snd_intel_sdw_acpi btbcm snd_hda_codec btmtk bluetooth uvcvideo snd_hda_core snd_usbmidi_lib snd_ump snd_rawmidi snd_hwdep snd_seq uvc videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_seq_device videobuf2_common snd_pcm videodev mei_hdcp mei_pxp iTCO_wdt at24 snd_timer mei_me intel_pmc_bxt iTCO_vendor_support mei snd rapl intel_cstate intel_uncore mc i2c_i801 lpc_ich rfkill i2c_smbus pcspkr soundcore apple_mfi_fastcharge joydev nfnetlink zram
Jul 13 04:02:26 localhost-live kernel:  lz4hc_compress lz4_compress overlay erofs netfs isofs hid_logitech_hidpp uas hid_logitech_dj usb_storage hid_apple amdgpu i915 amdxcp gpu_sched drm_panel_backlight_quirks drm_buddy drm_ttm_helper mxm_wmi ttm polyval_clmulni drm_exec polyval_generic ghash_clmulni_intel i2c_algo_bit drm_suballoc_helper sha512_ssse3 sha256_ssse3 drm_display_helper sha1_ssse3 nvme e1000e nvme_tcp cec nvme_fabrics video nvme_keyring wmi nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev
Jul 13 04:02:26 localhost-live kernel: CPU: 0 UID: 0 PID: 4785 Comm: mount Not tainted 6.14.0-63.fc42.x86_64 #1
Jul 13 04:02:26 localhost-live kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme4, BIOS P3.50 03/11/2018
Jul 13 04:02:26 localhost-live kernel: RIP: 0010:btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel: Code: 01 00 00 00 74 aa 0f 0b 48 83 bb 40 01 00 00 00 74 a8 0f 0b 48 83 bb 48 01 00 00 00 74 a6 0f 0b 48 83 bb 70 01 00 00 00 74 a4 <0f> 0b 48 83 bb 78 01 00 00 00 74 a2 0f 0b 48 83 bb a8 01 00 00 00
Jul 13 04:02:26 localhost-live kernel: RSP: 0018:ffffb1918a49bc38 EFLAGS: 00010286
Jul 13 04:02:26 localhost-live kernel: RAX: 0000000020000000 RBX: ffff9271822e3000 RCX: 0000000000000000
Jul 13 04:02:26 localhost-live kernel: RDX: 0000000020000000 RSI: ffff92723e622c00 RDI: ffff92723e622c08
Jul 13 04:02:26 localhost-live kernel: RBP: ffff927555127000 R08: ffff927555127000 R09: 0000000000100000
Jul 13 04:02:26 localhost-live kernel: R10: ffff9271c30575c8 R11: 0000000000000036 R12: ffff9271822e3090
Jul 13 04:02:26 localhost-live kernel: R13: ffff9271822e36c8 R14: 0000000000000000 R15: dead000000000100
Jul 13 04:02:26 localhost-live kernel: FS:  00007f67bbcda840(0000) GS:ffff927697600000(0000) knlGS:0000000000000000
Jul 13 04:02:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:02:26 localhost-live kernel: CR2: 00007fc484069000 CR3: 0000000175292002 CR4: 00000000001726f0
Jul 13 04:02:26 localhost-live kernel: Call Trace:
Jul 13 04:02:26 localhost-live kernel:  <TASK>
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_free_block_groups+0x34a/0x420
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? __warn.cold+0x93/0xfa
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? report_bug+0x103/0x150
Jul 13 04:02:26 localhost-live kernel:  ? handle_bug+0x58/0x90
Jul 13 04:02:26 localhost-live kernel:  ? exc_invalid_op+0x17/0x70
Jul 13 04:02:26 localhost-live kernel:  ? asm_exc_invalid_op+0x1a/0x20
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0x22/0xd0
Jul 13 04:02:26 localhost-live kernel:  btrfs_free_block_groups+0x34a/0x420
Jul 13 04:02:26 localhost-live kernel:  open_ctree+0xb40/0xbce
Jul 13 04:02:26 localhost-live kernel:  btrfs_get_tree_super.cold+0xb/0xbb
Jul 13 04:02:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:02:26 localhost-live kernel:  fc_mount+0x12/0x40
Jul 13 04:02:26 localhost-live kernel:  btrfs_get_tree_subvol+0x10d/0x210
Jul 13 04:02:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:02:26 localhost-live kernel:  vfs_cmd_create+0x59/0xe0
Jul 13 04:02:26 localhost-live kernel:  __do_sys_fsconfig+0x511/0x6f0
Jul 13 04:02:26 localhost-live kernel:  do_syscall_64+0x7f/0x170
Jul 13 04:02:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? do_readlinkat+0x12e/0x180
Jul 13 04:02:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? exc_page_fault+0x7e/0x1a0
Jul 13 04:02:26 localhost-live kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 13 04:02:26 localhost-live kernel: RIP: 0033:0x7f67bbeb9abe
Jul 13 04:02:26 localhost-live kernel: Code: 73 01 c3 48 8b 0d 4a 33 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1a 33 0f 00 f7 d8 64 89 01 48
Jul 13 04:02:26 localhost-live kernel: RSP: 002b:00007ffd51c98aa8 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
Quoted 9 lines
Jul 13 04:02:26 localhost-live kernel: RAX: ffffffffffffffda RBX: 000055ecc1aaa560 RCX: 00007f67bbeb9abe
Jul 13 04:02:26 localhost-live kernel: RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
Jul 13 04:02:26 localhost-live kernel: RBP: 00007ffd51c98bf0 R08: 0000000000000000 R09: 0000000000000003
Jul 13 04:02:26 localhost-live kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Jul 13 04:02:26 localhost-live kernel: R13: 000055ecc1aaaa00 R14: 00007f67bc036b00 R15: 000055ecc1aac498
Jul 13 04:02:26 localhost-live kernel:  </TASK>
Jul 13 04:02:26 localhost-live kernel: ---[ end trace 0000000000000000 ]---
Jul 13 04:02:26 localhost-live kernel: BTRFS error (device dm-1 state E): open_ctree failed: -5

Superblock dump collected with btrfs-tools 6.15 on live USB:

Quoted 99 lines

btrfs inspect-internal dump-super --full /dev/mapper/hogwarts-root
superblock: bytenr=65536, device=/dev/mapper/hogwarts-root
---------------------------------------------------------

csum_type               0 (crc32c)
csum_size               4
csum                    0x3e3b7bc2 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    a6999502-9362-4fa7-b7b4-28095790217d
metadata_uuid           00000000-0000-0000-0000-000000000000
label                   fedora_hogwarts
generation              456817
root                    202624811008
sys_array_size          97
chunk_root_generation   433110
root_level              0
chunk_root              368550690816
chunk_root_level        1
log_root                202650910720
log_root_transid (deprecated)   0
log_root_level          0
total_bytes             498382929920
bytes_used              227109515264
sectorsize              4096
nodesize                16384
leafsize (deprecated)   16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x3

( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID )
incompat_flags 0x171
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )

cache_generation        0
uuid_tree_generation    456817
dev_item.uuid           0721004b-8c3e-4ae7-bb3b-4b2bb5d313f7
dev_item.fsid           a6999502-9362-4fa7-b7b4-28095790217d [match]
dev_item.type           0
dev_item.total_bytes    498382929920
dev_item.bytes_used     253453402112
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 368550346752)
                length 33554432 owner 2 stripe_len 65536 type SYSTEM|single
                io_align 65536 io_width 65536 sector_size 4096
                num_stripes 1 sub_stripes 1
                        stripe 0 devid 1 offset 79512469504
                        dev_uuid 0721004b-8c3e-4ae7-bb3b-4b2bb5d313f7
backup_roots[4]:
        backup 0:
                backup_tree_root:       202624811008    gen: 456817     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202621485056    gen: 456817     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202621796352    gen: 456817     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227109515264
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       202407755776    gen: 456814     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202402742272    gen: 456814     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202403151872    gen: 456814     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227106684928
                backup_num_devices:     1

        backup 2:
                backup_tree_root:       202472538112    gen: 456815     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202447388672    gen: 456815     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202378985472    gen: 456815     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227107676160
                backup_num_devices:     1

        backup 3:
                backup_tree_root:       202539106304    gen: 456816     level: 0
Quoted 8 lines
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202537730048    gen: 456816     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202537861120    gen: 456816     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227108405248
                backup_num_devices:     1

Corrupt tree log dump collected with btrfs-tools 6.15 on live USB:

Quoted 25 lines
# btrfs inspect-internal dump-tree -t TREE_LOG /dev/mapper/hogwarts-root
btrfs-progs v6.15
log root tree
leaf 202650910720 items 2 free space 15355 generation 456818 owner TREE_LOG
leaf 202650910720 flags 0x1(WRITTEN) backref revision 1
fs uuid a6999502-9362-4fa7-b7b4-28095790217d
chunk uuid 710679da-fb76-4a51-9c84-8600a6e58c43
        item 0 key (TREE_LOG ROOT_ITEM 256) itemoff 15844 itemsize 439
                generation 456818 root_dirid 0 bytenr 202645028864 byte_limit 0 bytes_used 0
                last_snapshot 0 flags 0x0(none) refs 0
                drop_progress key (0 UNKNOWN.0 0) drop_level 0
                level 0 generation_v2 456818
                uuid 00000000-0000-0000-0000-000000000000
                parent_uuid 00000000-0000-0000-0000-000000000000
                received_uuid 00000000-0000-0000-0000-000000000000
                ctransid 0 otransid 0 stransid 0 rtransid 0
                ctime 0.0 (1970-01-01 00:00:00)
                otime 0.0 (1970-01-01 00:00:00)
                stime 0.0 (1970-01-01 00:00:00)
                rtime 0.0 (1970-01-01 00:00:00)
        item 1 key (TREE_LOG ROOT_ITEM 258) itemoff 15405 itemsize 439
                generation 456818 root_dirid 0 bytenr 202650877952 byte_limit 0 bytes_used 65536
                last_snapshot 0 flags 0x0(none) refs 0
                drop_progress key (0 UNKNOWN.0 0) drop_level 0
                level 1 generation_v2 456818
Quoted 8 lines
                uuid 00000000-0000-0000-0000-000000000000
                parent_uuid 00000000-0000-0000-0000-000000000000
                received_uuid 00000000-0000-0000-0000-000000000000
                ctransid 0 otransid 0 stransid 0 rtransid 0
                ctime 0.0 (1970-01-01 00:00:00)
                otime 0.0 (1970-01-01 00:00:00)
                stime 0.0 (1970-01-01 00:00:00)
                rtime 0.0 (1970-01-01 00:00:00)

If I'm interpreting correctly, the corrupt tree log is zeroed, so that kind of gestures in the direction of what the log replay code might be mishandling.
[1]
https://discussion.fedoraproject.org/t/btrfs-error-emergency-mode-entered-every-time-i-boot/141179

Thanks,

Russell

#2 lore raw
Quoted 7 lines · Russell Haley

Gentle ping.

@Filipe You've recently submitted a cleanup of the log replay code, so I assume you're familiar with it. Is there anything Funny Lookin' in the path that handles a corrupt TREE_LOG? Would've changed sometime between 6.14.9 and 6.15.4.

No need to ping, I'm well aware of the issue, it has been reported by other people in the mailing list, IRC, bugzillas, etc. I've been looking into the issue, but haven't found yet the cause.

Thanks.

Quoted 358 lines

Thanks,

Russell

Quoted 350 lines · Russell Haley

Howdy,

After a power outage, my computer booted into emergency mode, apparently due to a corrupt log tree on the root SSD (WD Blue, WDC WDS500G2B0C-00PXH0). Symptoms basically identical to [1], and I was eventually able to fix it the same way, with btrfs rescue zero-log.

*However*, troubleshooting was complicated by the fact that, on kernel 6.15.4, attempting to mount the corrupt FS even once OOPSed the kernel and locked the device, making it impossible to run btrfs commands against it or try different mount options. This would've been a showstopper if I didn't have Fedora's backup kernels to try and see that the dmesg error was different. That led me to boot into a live USB with kernel 6.14.0, from which I was able to investigate. I then booted into backup kernel 6.14.9 (the newest non-problematic) and zeroed the log.

Facts:

1. Kernel 6.15.4 has the null deref.

2. Kernel 6.14.9 does not.

3. mount -o ro,rescue=nologreplay succeeds on kernel 6.14.0 (from live USB) and 6.15.4 (newer live USB). On 6.15.4, it succeeds if it is the first mount command run.

Here's all the evidence I collected before zeroing the log so I could have a working PC again. It's the log from the bad kernel, one good kernel, and btrfs inspect-internal dump-super and dump-tree of the corrupted disk

Log from kernel 6.15.4:

Quoted 81 lines
Jul 13 04:25:26 localhost-live kernel: BTRFS: device label fedora_hogwarts devid 1 transid 456817 /dev/mapper/hogwarts-root (252:1) scanned by mount (4794)
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): first mount of filesystem a6999502-9362-4fa7-b7b4-28095790217d
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): using crc32c (crc32c-x86_64) checksum algorithm
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): using free-space-tree
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): bdev /dev/mapper/hogwarts-root errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Jul 13 04:25:26 localhost-live kernel: BTRFS info (device dm-1): start tree-log replay
Jul 13 04:25:26 localhost-live kernel: BUG: kernel NULL pointer dereference, address: 0000000000000219
Jul 13 04:25:26 localhost-live kernel: #PF: supervisor read access in kernel mode
Jul 13 04:25:26 localhost-live kernel: #PF: error_code(0x0000) - not-present page
Jul 13 04:25:26 localhost-live kernel: PGD 0 P4D 0
Jul 13 04:25:26 localhost-live kernel: Oops: Oops: 0000 [#1] SMP PTI
Jul 13 04:25:26 localhost-live kernel: CPU: 0 UID: 0 PID: 4794 Comm: mount Not tainted 6.15.4-200.fc42.x86_64 #1 PREEMPT(lazy)
Jul 13 04:25:26 localhost-live kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme4, BIOS P3.50 03/11/2018
Jul 13 04:25:26 localhost-live kernel: RIP: 0010:iput+0x20/0x230
Jul 13 04:25:26 localhost-live kernel: Code: 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 86 01 00 00 41 54 55 48 8d af 50 01 00 00 53 48 89 fb <f6> 87 91 00 00 00 01 74 3a e9 68 01 00 00 8b 53 48 8b 83 90 00 00
Jul 13 04:25:26 localhost-live kernel: RSP: 0018:ffffd43c0b893740 EFLAGS: 00010206
Jul 13 04:25:26 localhost-live kernel: RAX: 0000000000000000 RBX: 0000000000000188 RCX: 000000000092a000
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: ffffffff93bcea60 RDI: 0000000000000188
Jul 13 04:25:26 localhost-live kernel: RBP: 00000000000002d8 R08: 0000000000000000 R09: ffffffff90db25b6
Jul 13 04:25:26 localhost-live kernel: R10: ffff8c4aa9480000 R11: fffff60205a52000 R12: ffffd43c0b8938bf
Jul 13 04:25:26 localhost-live kernel: R13: 0000000000000000 R14: ffff8c4aa9476690 R15: 00000000fffffffb
Jul 13 04:25:26 localhost-live kernel: FS:  00007f1b5f3f9840(0000) GS:ffff8c4fc3a71000(0000) knlGS:0000000000000000
Jul 13 04:25:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219 CR3: 000000016abc8003 CR4: 00000000001726f0
Jul 13 04:25:26 localhost-live kernel: Call Trace:
Jul 13 04:25:26 localhost-live kernel:  <TASK>
Jul 13 04:25:26 localhost-live kernel:  replay_one_extent+0xba/0x740
Jul 13 04:25:26 localhost-live kernel:  ? release_extent_buffer+0xb5/0xd0
Jul 13 04:25:26 localhost-live kernel:  ? copy_extent_buffer+0x126/0x160
Jul 13 04:25:26 localhost-live kernel:  ? btrfs_release_path+0x2b/0x1b0
Jul 13 04:25:26 localhost-live kernel:  replay_one_buffer+0x2ee/0x500
Jul 13 04:25:26 localhost-live kernel:  ? alloc_extent_buffer+0x97/0x500
Jul 13 04:25:26 localhost-live kernel:  ? read_extent_buffer+0x102/0x140
Jul 13 04:25:26 localhost-live kernel:  walk_down_log_tree+0x1d8/0x4c0
Jul 13 04:25:26 localhost-live kernel:  ? kmem_cache_alloc_noprof+0x141/0x410
Jul 13 04:25:26 localhost-live kernel:  walk_log_tree+0xe6/0x280
Jul 13 04:25:26 localhost-live kernel:  btrfs_recover_log_trees+0x1cc/0x5a0
Jul 13 04:25:26 localhost-live kernel:  ? __pfx_replay_one_buffer+0x10/0x10
Jul 13 04:25:26 localhost-live kernel:  open_ctree+0x912/0xb98
Jul 13 04:25:26 localhost-live kernel:  btrfs_get_tree_super.cold+0xb/0xbf
Jul 13 04:25:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:25:26 localhost-live kernel:  fc_mount+0x12/0x40
Jul 13 04:25:26 localhost-live kernel:  btrfs_get_tree_subvol+0x10d/0x210
Jul 13 04:25:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:25:26 localhost-live kernel:  vfs_cmd_create+0x57/0xd0
Jul 13 04:25:26 localhost-live kernel:  __do_sys_fsconfig+0x4b1/0x640
Jul 13 04:25:26 localhost-live kernel:  do_syscall_64+0x7b/0x160
Jul 13 04:25:26 localhost-live kernel:  ? __check_object_size.part.0+0x59/0xc0
Jul 13 04:25:26 localhost-live kernel:  ? _copy_from_user+0x27/0x60
Jul 13 04:25:26 localhost-live kernel:  ? lookup_constant+0x35/0x50
Jul 13 04:25:26 localhost-live kernel:  ? vfs_parse_fs_param+0x30/0x100
Jul 13 04:25:26 localhost-live kernel:  ? __do_sys_fsconfig+0x354/0x640
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? __do_sys_newfstatat+0x4b/0x80
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? do_syscall_64+0x87/0x160
Jul 13 04:25:26 localhost-live kernel:  ? exc_page_fault+0x7e/0x1a0
Jul 13 04:25:26 localhost-live kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 13 04:25:26 localhost-live kernel: RIP: 0033:0x7f1b5f5d7abe
Jul 13 04:25:26 localhost-live kernel: Code: 73 01 c3 48 8b 0d 4a 33 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1a 33 0f 00 f7 d8 64 89 01 48
Jul 13 04:25:26 localhost-live kernel: RSP: 002b:00007ffe53604898 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
Jul 13 04:25:26 localhost-live kernel: RAX: ffffffffffffffda RBX: 0000556b3a8eb560 RCX: 00007f1b5f5d7abe
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
Jul 13 04:25:26 localhost-live kernel: RBP: 00007ffe536049e0 R08: 0000000000000000 R09: 0000000000000003
Jul 13 04:25:26 localhost-live kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Jul 13 04:25:26 localhost-live kernel: R13: 0000556b3a8eba00 R14: 00007f1b5f754b00 R15: 0000556b3a8ed498
Jul 13 04:25:26 localhost-live kernel:  </TASK>
Jul 13 04:25:26 localhost-live kernel: Modules linked in: f2fs dm_crypt uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi bnep snd_hda_scodec_component snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi btusb snd_hda_codec btrtl btintel intel_rapl_msr snd_hda_core btbcm uvcvideo intel_rapl_common snd_usb_audio btmtk snd_usbmidi_lib snd_ump snd_rawmidi x86_pkg_temp_thermal snd_hwdep intel_powerclamp bluetooth snd_seq uvc videobuf2_vmalloc coretemp videobuf2_memops videobuf2_v4l2 kvm_intel snd_seq_device videobuf2_common snd_pcm videodev mei_hdcp mei_pxp iTCO_wdt intel_pmc_bxt at24 snd_timer mei_me iTCO_vendor_support kvm snd mei mc i2c_i801 joydev i2c_smbus pcspkr lpc_ich rfkill apple_mfi_fastcharge irqbypass rapl intel_cstate intel_uncore soundcore nfnetlink
Jul 13 04:25:26 localhost-live kernel:  zram lz4hc_compress lz4_compress overlay erofs netfs hid_logitech_hidpp isofs amdgpu hid_logitech_dj i915 mxm_wmi amdxcp polyval_clmulni gpu_sched polyval_generic drm_panel_backlight_quirks ghash_clmulni_intel drm_ttm_helper drm_buddy sha512_ssse3 ttm drm_exec sha256_ssse3 i2c_algo_bit uas drm_suballoc_helper sha1_ssse3 hid_apple usb_storage drm_display_helper e1000e nvme cec nvme_tcp nvme_fabrics nvme_core video wmi nvme_keyring nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219
Jul 13 04:25:26 localhost-live kernel: ---[ end trace 0000000000000000 ]---
Jul 13 04:25:26 localhost-live kernel: RIP: 0010:iput+0x20/0x230
Jul 13 04:25:26 localhost-live kernel: Code: 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 85 ff 0f 84 86 01 00 00 41 54 55 48 8d af 50 01 00 00 53 48 89 fb <f6> 87 91 00 00 00 01 74 3a e9 68 01 00 00 8b 53 48 8b 83 90 00 00
Jul 13 04:25:26 localhost-live kernel: RSP: 0018:ffffd43c0b893740 EFLAGS: 00010206
Quoted 8 lines
Jul 13 04:25:26 localhost-live kernel: RAX: 0000000000000000 RBX: 0000000000000188 RCX: 000000000092a000
Jul 13 04:25:26 localhost-live kernel: RDX: 0000000000000000 RSI: ffffffff93bcea60 RDI: 0000000000000188
Jul 13 04:25:26 localhost-live kernel: RBP: 00000000000002d8 R08: 0000000000000000 R09: ffffffff90db25b6
Jul 13 04:25:26 localhost-live kernel: R10: ffff8c4aa9480000 R11: fffff60205a52000 R12: ffffd43c0b8938bf
Jul 13 04:25:26 localhost-live kernel: R13: 0000000000000000 R14: ffff8c4aa9476690 R15: 00000000fffffffb
Jul 13 04:25:26 localhost-live kernel: FS:  00007f1b5f3f9840(0000) GS:ffff8c4fc3a71000(0000) knlGS:0000000000000000
Jul 13 04:25:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:25:26 localhost-live kernel: CR2: 0000000000000219 CR3: 000000016abc8003 CR4: 00000000001726f0

Log from kernel 6.14.0:

Quoted 60 lines
Jul 13 04:02:25 localhost-live kernel: BTRFS: device label fedora_hogwarts devid 1 transid 456817 /dev/mapper/hogwarts-root (252:1) scanned by mount (4785)
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): first mount of filesystem a6999502-9362-4fa7-b7b4-28095790217d
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): using crc32c (crc32c-x86_64) checksum algorithm
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): using free-space-tree
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): bdev /dev/mapper/hogwarts-root errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Jul 13 04:02:25 localhost-live kernel: BTRFS info (device dm-1): start tree-log replay
Jul 13 04:02:26 localhost-live kernel: BTRFS: error (device dm-1) in btrfs_replay_log:2104: errno=-5 IO failure (Failed to recover log tree)
Jul 13 04:02:26 localhost-live kernel: ------------[ cut here ]------------
Jul 13 04:02:26 localhost-live kernel: WARNING: CPU: 0 PID: 4785 at fs/btrfs/block-rsv.c:452 btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel: Modules linked in: dm_crypt uinput rfcomm snd_seq_dummy snd_hrtimer nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp qrtr coretemp kvm_intel snd_hda_codec_realtek snd_hda_codec_generic bnep snd_hda_codec_hdmi kvm snd_usb_audio btusb snd_hda_scodec_component snd_hda_intel btrtl btintel snd_intel_dspcfg snd_intel_sdw_acpi btbcm snd_hda_codec btmtk bluetooth uvcvideo snd_hda_core snd_usbmidi_lib snd_ump snd_rawmidi snd_hwdep snd_seq uvc videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 snd_seq_device videobuf2_common snd_pcm videodev mei_hdcp mei_pxp iTCO_wdt at24 snd_timer mei_me intel_pmc_bxt iTCO_vendor_support mei snd rapl intel_cstate intel_uncore mc i2c_i801 lpc_ich rfkill i2c_smbus pcspkr soundcore apple_mfi_fastcharge joydev nfnetlink zram
Jul 13 04:02:26 localhost-live kernel:  lz4hc_compress lz4_compress overlay erofs netfs isofs hid_logitech_hidpp uas hid_logitech_dj usb_storage hid_apple amdgpu i915 amdxcp gpu_sched drm_panel_backlight_quirks drm_buddy drm_ttm_helper mxm_wmi ttm polyval_clmulni drm_exec polyval_generic ghash_clmulni_intel i2c_algo_bit drm_suballoc_helper sha512_ssse3 sha256_ssse3 drm_display_helper sha1_ssse3 nvme e1000e nvme_tcp cec nvme_fabrics video nvme_keyring wmi nvme_core nvme_auth sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi loop fuse i2c_dev
Jul 13 04:02:26 localhost-live kernel: CPU: 0 UID: 0 PID: 4785 Comm: mount Not tainted 6.14.0-63.fc42.x86_64 #1
Jul 13 04:02:26 localhost-live kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme4, BIOS P3.50 03/11/2018
Jul 13 04:02:26 localhost-live kernel: RIP: 0010:btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel: Code: 01 00 00 00 74 aa 0f 0b 48 83 bb 40 01 00 00 00 74 a8 0f 0b 48 83 bb 48 01 00 00 00 74 a6 0f 0b 48 83 bb 70 01 00 00 00 74 a4 <0f> 0b 48 83 bb 78 01 00 00 00 74 a2 0f 0b 48 83 bb a8 01 00 00 00
Jul 13 04:02:26 localhost-live kernel: RSP: 0018:ffffb1918a49bc38 EFLAGS: 00010286
Jul 13 04:02:26 localhost-live kernel: RAX: 0000000020000000 RBX: ffff9271822e3000 RCX: 0000000000000000
Jul 13 04:02:26 localhost-live kernel: RDX: 0000000020000000 RSI: ffff92723e622c00 RDI: ffff92723e622c08
Jul 13 04:02:26 localhost-live kernel: RBP: ffff927555127000 R08: ffff927555127000 R09: 0000000000100000
Jul 13 04:02:26 localhost-live kernel: R10: ffff9271c30575c8 R11: 0000000000000036 R12: ffff9271822e3090
Jul 13 04:02:26 localhost-live kernel: R13: ffff9271822e36c8 R14: 0000000000000000 R15: dead000000000100
Jul 13 04:02:26 localhost-live kernel: FS:  00007f67bbcda840(0000) GS:ffff927697600000(0000) knlGS:0000000000000000
Jul 13 04:02:26 localhost-live kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 13 04:02:26 localhost-live kernel: CR2: 00007fc484069000 CR3: 0000000175292002 CR4: 00000000001726f0
Jul 13 04:02:26 localhost-live kernel: Call Trace:
Jul 13 04:02:26 localhost-live kernel:  <TASK>
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? show_trace_log_lvl+0x1d3/0x300
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_free_block_groups+0x34a/0x420
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? __warn.cold+0x93/0xfa
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? report_bug+0x103/0x150
Jul 13 04:02:26 localhost-live kernel:  ? handle_bug+0x58/0x90
Jul 13 04:02:26 localhost-live kernel:  ? exc_invalid_op+0x17/0x70
Jul 13 04:02:26 localhost-live kernel:  ? asm_exc_invalid_op+0x1a/0x20
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0xb0/0xd0
Jul 13 04:02:26 localhost-live kernel:  ? btrfs_release_global_block_rsv+0x22/0xd0
Jul 13 04:02:26 localhost-live kernel:  btrfs_free_block_groups+0x34a/0x420
Jul 13 04:02:26 localhost-live kernel:  open_ctree+0xb40/0xbce
Jul 13 04:02:26 localhost-live kernel:  btrfs_get_tree_super.cold+0xb/0xbb
Jul 13 04:02:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:02:26 localhost-live kernel:  fc_mount+0x12/0x40
Jul 13 04:02:26 localhost-live kernel:  btrfs_get_tree_subvol+0x10d/0x210
Jul 13 04:02:26 localhost-live kernel:  vfs_get_tree+0x29/0xd0
Jul 13 04:02:26 localhost-live kernel:  vfs_cmd_create+0x59/0xe0
Jul 13 04:02:26 localhost-live kernel:  __do_sys_fsconfig+0x511/0x6f0
Jul 13 04:02:26 localhost-live kernel:  do_syscall_64+0x7f/0x170
Jul 13 04:02:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? do_readlinkat+0x12e/0x180
Jul 13 04:02:26 localhost-live kernel:  ? syscall_exit_to_user_mode+0x10/0x210
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? do_syscall_64+0x8c/0x170
Jul 13 04:02:26 localhost-live kernel:  ? exc_page_fault+0x7e/0x1a0
Jul 13 04:02:26 localhost-live kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jul 13 04:02:26 localhost-live kernel: RIP: 0033:0x7f67bbeb9abe
Jul 13 04:02:26 localhost-live kernel: Code: 73 01 c3 48 8b 0d 4a 33 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1a 33 0f 00 f7 d8 64 89 01 48
Jul 13 04:02:26 localhost-live kernel: RSP: 002b:00007ffd51c98aa8 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
Quoted 9 lines
Jul 13 04:02:26 localhost-live kernel: RAX: ffffffffffffffda RBX: 000055ecc1aaa560 RCX: 00007f67bbeb9abe
Jul 13 04:02:26 localhost-live kernel: RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
Jul 13 04:02:26 localhost-live kernel: RBP: 00007ffd51c98bf0 R08: 0000000000000000 R09: 0000000000000003
Jul 13 04:02:26 localhost-live kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Jul 13 04:02:26 localhost-live kernel: R13: 000055ecc1aaaa00 R14: 00007f67bc036b00 R15: 000055ecc1aac498
Jul 13 04:02:26 localhost-live kernel:  </TASK>
Jul 13 04:02:26 localhost-live kernel: ---[ end trace 0000000000000000 ]---
Jul 13 04:02:26 localhost-live kernel: BTRFS error (device dm-1 state E): open_ctree failed: -5

Superblock dump collected with btrfs-tools 6.15 on live USB:

Quoted 99 lines

btrfs inspect-internal dump-super --full /dev/mapper/hogwarts-root
superblock: bytenr=65536, device=/dev/mapper/hogwarts-root
---------------------------------------------------------

csum_type               0 (crc32c)
csum_size               4
csum                    0x3e3b7bc2 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    a6999502-9362-4fa7-b7b4-28095790217d
metadata_uuid           00000000-0000-0000-0000-000000000000
label                   fedora_hogwarts
generation              456817
root                    202624811008
sys_array_size          97
chunk_root_generation   433110
root_level              0
chunk_root              368550690816
chunk_root_level        1
log_root                202650910720
log_root_transid (deprecated)   0
log_root_level          0
total_bytes             498382929920
bytes_used              227109515264
sectorsize              4096
nodesize                16384
leafsize (deprecated)   16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x0
compat_ro_flags         0x3

( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID )
incompat_flags 0x171
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )

cache_generation        0
uuid_tree_generation    456817
dev_item.uuid           0721004b-8c3e-4ae7-bb3b-4b2bb5d313f7
dev_item.fsid           a6999502-9362-4fa7-b7b4-28095790217d [match]
dev_item.type           0
dev_item.total_bytes    498382929920
dev_item.bytes_used     253453402112
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 368550346752)
                length 33554432 owner 2 stripe_len 65536 type SYSTEM|single
                io_align 65536 io_width 65536 sector_size 4096
                num_stripes 1 sub_stripes 1
                        stripe 0 devid 1 offset 79512469504
                        dev_uuid 0721004b-8c3e-4ae7-bb3b-4b2bb5d313f7
backup_roots[4]:
        backup 0:
                backup_tree_root:       202624811008    gen: 456817     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202621485056    gen: 456817     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202621796352    gen: 456817     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227109515264
                backup_num_devices:     1

        backup 1:
                backup_tree_root:       202407755776    gen: 456814     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202402742272    gen: 456814     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202403151872    gen: 456814     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227106684928
                backup_num_devices:     1

        backup 2:
                backup_tree_root:       202472538112    gen: 456815     level: 0
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202447388672    gen: 456815     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202378985472    gen: 456815     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227107676160
                backup_num_devices:     1

        backup 3:
                backup_tree_root:       202539106304    gen: 456816     level: 0
Quoted 8 lines
                backup_chunk_root:      368550690816    gen: 433110     level: 1
                backup_extent_root:     202537730048    gen: 456816     level: 2
                backup_fs_root:         29755916288     gen: 36186      level: 0
                backup_dev_root:        1283227648      gen: 453657     level: 1
                csum_root:      202537861120    gen: 456816     level: 2
                backup_total_bytes:     498382929920
                backup_bytes_used:      227108405248
                backup_num_devices:     1

Corrupt tree log dump collected with btrfs-tools 6.15 on live USB:

Quoted 25 lines
# btrfs inspect-internal dump-tree -t TREE_LOG /dev/mapper/hogwarts-root
btrfs-progs v6.15
log root tree
leaf 202650910720 items 2 free space 15355 generation 456818 owner TREE_LOG
leaf 202650910720 flags 0x1(WRITTEN) backref revision 1
fs uuid a6999502-9362-4fa7-b7b4-28095790217d
chunk uuid 710679da-fb76-4a51-9c84-8600a6e58c43
        item 0 key (TREE_LOG ROOT_ITEM 256) itemoff 15844 itemsize 439
                generation 456818 root_dirid 0 bytenr 202645028864 byte_limit 0 bytes_used 0
                last_snapshot 0 flags 0x0(none) refs 0
                drop_progress key (0 UNKNOWN.0 0) drop_level 0
                level 0 generation_v2 456818
                uuid 00000000-0000-0000-0000-000000000000
                parent_uuid 00000000-0000-0000-0000-000000000000
                received_uuid 00000000-0000-0000-0000-000000000000
                ctransid 0 otransid 0 stransid 0 rtransid 0
                ctime 0.0 (1970-01-01 00:00:00)
                otime 0.0 (1970-01-01 00:00:00)
                stime 0.0 (1970-01-01 00:00:00)
                rtime 0.0 (1970-01-01 00:00:00)
        item 1 key (TREE_LOG ROOT_ITEM 258) itemoff 15405 itemsize 439
                generation 456818 root_dirid 0 bytenr 202650877952 byte_limit 0 bytes_used 65536
                last_snapshot 0 flags 0x0(none) refs 0
                drop_progress key (0 UNKNOWN.0 0) drop_level 0
                level 1 generation_v2 456818
Quoted 8 lines
                uuid 00000000-0000-0000-0000-000000000000
                parent_uuid 00000000-0000-0000-0000-000000000000
                received_uuid 00000000-0000-0000-0000-000000000000
                ctransid 0 otransid 0 stransid 0 rtransid 0
                ctime 0.0 (1970-01-01 00:00:00)
                otime 0.0 (1970-01-01 00:00:00)
                stime 0.0 (1970-01-01 00:00:00)
                rtime 0.0 (1970-01-01 00:00:00)

If I'm interpreting correctly, the corrupt tree log is zeroed, so that kind of gestures in the direction of what the log replay code might be mishandling.
[1]
https://discussion.fedoraproject.org/t/btrfs-error-emergency-mode-entered-every-time-i-boot/141179

Thanks,

Russell

#3 lore raw
In reply to Russell Haley
Quoted 15 lines · Russell Haley

Howdy,

After a power outage, my computer booted into emergency mode, apparently due to a corrupt log tree on the root SSD (WD Blue, WDC WDS500G2B0C-00PXH0). Symptoms basically identical to [1], and I was eventually able to fix it the same way, with btrfs rescue zero-log.

*However*, troubleshooting was complicated by the fact that, on kernel 6.15.4, attempting to mount the corrupt FS even once OOPSed the kernel and locked the device, making it impossible to run btrfs commands against it or try different mount options. This would've been a showstopper if I didn't have Fedora's backup kernels to try and see that the dmesg error was different. That led me to boot into a live USB with kernel 6.14.0, from which I was able to investigate. I then booted into backup kernel 6.14.9 (the newest non-problematic) and zeroed the log.

[..]

Quoted 36 lines

Corrupt tree log dump collected with btrfs-tools 6.15 on live USB:

Quoted 25 lines
# btrfs inspect-internal dump-tree -t TREE_LOG /dev/mapper/hogwarts-root
btrfs-progs v6.15
log root tree
leaf 202650910720 items 2 free space 15355 generation 456818 owner TREE_LOG
leaf 202650910720 flags 0x1(WRITTEN) backref revision 1
fs uuid a6999502-9362-4fa7-b7b4-28095790217d
chunk uuid 710679da-fb76-4a51-9c84-8600a6e58c43
         item 0 key (TREE_LOG ROOT_ITEM 256) itemoff 15844 itemsize 439
                 generation 456818 root_dirid 0 bytenr 202645028864 byte_limit 0 bytes_used 0
                 last_snapshot 0 flags 0x0(none) refs 0
                 drop_progress key (0 UNKNOWN.0 0) drop_level 0
                 level 0 generation_v2 456818
                 uuid 00000000-0000-0000-0000-000000000000
                 parent_uuid 00000000-0000-0000-0000-000000000000
                 received_uuid 00000000-0000-0000-0000-000000000000
                 ctransid 0 otransid 0 stransid 0 rtransid 0
                 ctime 0.0 (1970-01-01 00:00:00)
                 otime 0.0 (1970-01-01 00:00:00)
                 stime 0.0 (1970-01-01 00:00:00)
                 rtime 0.0 (1970-01-01 00:00:00)
         item 1 key (TREE_LOG ROOT_ITEM 258) itemoff 15405 itemsize 439
                 generation 456818 root_dirid 0 bytenr 202650877952 byte_limit 0 bytes_used 65536
                 last_snapshot 0 flags 0x0(none) refs 0
                 drop_progress key (0 UNKNOWN.0 0) drop_level 0
                 level 1 generation_v2 456818
Quoted 8 lines
                 uuid 00000000-0000-0000-0000-000000000000
                 parent_uuid 00000000-0000-0000-0000-000000000000
                 received_uuid 00000000-0000-0000-0000-000000000000
                 ctransid 0 otransid 0 stransid 0 rtransid 0
                 ctime 0.0 (1970-01-01 00:00:00)
                 otime 0.0 (1970-01-01 00:00:00)
                 stime 0.0 (1970-01-01 00:00:00)
                 rtime 0.0 (1970-01-01 00:00:00)
Quoted 3 lines

If I'm interpreting correctly, the corrupt tree log is zeroed, so that kind of gestures in the direction of what the log replay code might be mishandling.

Hope you still have the fs at hand.

In that case, I believe the following dump would help Filipe to pin down the problem:

# btrfs ins dump-tree -b 202645028864 --follow <device>
# btrfs ins dump-tree -b 202650877952 --follow <device>
# btrfs ins dump-tree -t 256 <device>
# btrfs ins dump-tree -t 258 <device>

Log tree replay is a very complex mechanism, it records changes into the log tree (dumped by the first two commands). And replaying the log means we need to use the difference (the log tree) to update the target subvolumes (dumped by the last two commands).

Problems in either tree can cause replay failure, thus you need to dump both the log trees (not only the log tree root, but the real log tree for each subvolume) and the source subvolumes.

Thanks,
Qu

Quoted 7 lines

[1]
https://discussion.fedoraproject.org/t/btrfs-error-emergency-mode-entered-every-time-i-boot/141179

Thanks,

Russell

#4 lore raw
Quoted 1 line · Qu Wenruo

Hope you still have the fs at hand.

I do, but I didn't image it, and it's been in active use for the last week. I've dumped what you suggested while running kernel 6.15.5.

Quoted 5 lines

In that case, I believe the following dump would help Filipe to pin down the problem:

# btrfs ins dump-tree -b 202645028864 --follow <device>
# btrfs ins dump-tree -b 202650877952 --follow <device>

These begin with

Quoted 1 line

leaf 202645028864 items 358 free space 329 generation 460240 owner CSUM_TREE

and

Quoted 1 line

leaf 202650877952 items 130 free space 6143 generation 460240 owner EXTENT_TREE

Owner not being LOG_TREE means those blocks have been reused and the evidence is gone? Attached just in case it's still useful, but I suspect not.

Quoted 2 lines
# btrfs ins dump-tree -t 256 <device>
# btrfs ins dump-tree -t 258 <device>

These are quite large. The first is 17 MB and the second is 1.9 GB. Not attached, due to size and likelihood of being useless.

I'm not going to start fipping the power switch to corrupt my disk intentionally, but if it happens again, I'll collect these dumps in the immediate aftermath. I assume you formed those commands from the ROOT_ITEM and bytenr in the -t TREE_LOG dump.

Quoted 8 lines

Log tree replay is a very complex mechanism, it records changes into the log tree (dumped by the first two commands).
And replaying the log means we need to use the difference (the log tree) to update the target subvolumes (dumped by the last two commands).

Problems in either tree can cause replay failure, thus you need to dump both the log trees (not only the log tree root, but the real log tree for each subvolume) and the source subvolumes.

It seems like there are two issues at play here. I posted to the list because the 6.15 kernel handled replay failure ungracefully, leaving the block device in an uninteractable state.

Until I saw the other threads about a spike in log tree corruption reports, I had assumed the corruption itself was caused by hardware misbehaving during uncontrolled mains voltage brownout.

Thanks,

Russell

btrfs-progs v6.15
leaf 202645028864 items 358 free space 329 generation 460240 owner CSUM_TREE leaf 202645028864 flags 0x1(WRITTEN) backref revision 1
fs uuid a6999502-9362-4fa7-b7b4-28095790217d
chunk uuid 710679da-fb76-4a51-9c84-8600a6e58c43

	item 0 key (EXTENT_CSUM EXTENT_CSUM 7225188352) itemoff 16263 itemsize 20
		range start 7225188352 end 7225208832 length 20480
	item 1 key (EXTENT_CSUM EXTENT_CSUM 7225208832) itemoff 16247 itemsize 16
		range start 7225208832 end 7225225216 length 16384
	item 2 key (EXTENT_CSUM EXTENT_CSUM 7225225216) itemoff 16243 itemsize 4
		range start 7225225216 end 7225229312 length 4096
	item 3 key (EXTENT_CSUM EXTENT_CSUM 7225229312) itemoff 16227 itemsize 16
		range start 7225229312 end 7225245696 length 16384
	item 4 key (EXTENT_CSUM EXTENT_CSUM 7225245696) itemoff 16195 itemsize 32
		range start 7225245696 end 7225278464 length 32768
	item 5 key (EXTENT_CSUM EXTENT_CSUM 7225278464) itemoff 16159 itemsize 36
		range start 7225278464 end 7225315328 length 36864
	item 6 key (EXTENT_CSUM EXTENT_CSUM 7225315328) itemoff 16143 itemsize 16
		range start 7225315328 end 7225331712 length 16384
	item 7 key (EXTENT_CSUM EXTENT_CSUM 7225331712) itemoff 16035 itemsize 108
		range start 7225331712 end 7225442304 length 110592
	item 8 key (EXTENT_CSUM EXTENT_CSUM 7225442304) itemoff 16031 itemsize 4
		range start 7225442304 end 7225446400 length 4096
	item 9 key (EXTENT_CSUM EXTENT_CSUM 7225446400) itemoff 16003 itemsize 28
		range start 7225446400 end 7225475072 length 28672
	item 10 key (EXTENT_CSUM EXTENT_CSUM 7225475072) itemoff 15967 itemsize 36
		range start 7225475072 end 7225511936 length 36864
	item 11 key (EXTENT_CSUM EXTENT_CSUM 7225511936) itemoff 15959 itemsize 8
		range start 7225511936 end 7225520128 length 8192
	item 12 key (EXTENT_CSUM EXTENT_CSUM 7225520128) itemoff 15955 itemsize 4
		range start 7225520128 end 7225524224 length 4096
	item 13 key (EXTENT_CSUM EXTENT_CSUM 7225524224) itemoff 15927 itemsize 28
		range start 7225524224 end 7225552896 length 28672
	item 14 key (EXTENT_CSUM EXTENT_CSUM 7225552896) itemoff 15911 itemsize 16
		range start 7225552896 end 7225569280 length 16384
	item 15 key (EXTENT_CSUM EXTENT_CSUM 7225569280) itemoff 15907 itemsize 4
		range start 7225569280 end 7225573376 length 4096
	item 16 key (EXTENT_CSUM EXTENT_CSUM 7225573376) itemoff 15871 itemsize 36
		range start 7225573376 end 7225610240 length 36864
	item 17 key (EXTENT_CSUM EXTENT_CSUM 7225610240) itemoff 15847 itemsize 24
		range start 7225610240 end 7225634816 length 24576
	item 18 key (EXTENT_CSUM EXTENT_CSUM 7225634816) itemoff 15843 itemsize 4
		range start 7225634816 end 7225638912 length 4096
	item 19 key (EXTENT_CSUM EXTENT_CSUM 7225638912) itemoff 15819 itemsize 24
		range start 7225638912 end 7225663488 length 24576
	item 20 key (EXTENT_CSUM EXTENT_CSUM 7225663488) itemoff 15815 itemsize 4
		range start 7225663488 end 7225667584 length 4096
	item 21 key (EXTENT_CSUM EXTENT_CSUM 7225667584) itemoff 15795 itemsize 20
		range start 7225667584 end 7225688064 length 20480
	item 22 key (EXTENT_CSUM EXTENT_CSUM 7225688064) itemoff 15791 itemsize 4
		range start 7225688064 end 7225692160 length 4096
	item 23 key (EXTENT_CSUM EXTENT_CSUM 7225692160) itemoff 15783 itemsize 8
		range start 7225692160 end 7225700352 length 8192
	item 24 key (EXTENT_CSUM EXTENT_CSUM 7225700352) itemoff 15771 itemsize 12
		range start 7225700352 end 7225712640 length 12288
	item 25 key (EXTENT_CSUM EXTENT_CSUM 7225712640) itemoff 15763 itemsize 8
		range start 7225712640 end 7225720832 length 8192
	item 26 key (EXTENT_CSUM EXTENT_CSUM 7225720832) itemoff 15711 itemsize 52
		range start 7225720832 end 7225774080 length 53248
	item 27 key (EXTENT_CSUM EXTENT_CSUM 7225774080) itemoff 15703 itemsize 8
		range start 7225774080 end 7225782272 length 8192
	item 28 key (EXTENT_CSUM EXTENT_CSUM 7225782272) itemoff 15699 itemsize 4
		range start 7225782272 end 7225786368 length 4096
	item 29 key (EXTENT_CSUM EXTENT_CSUM 7225786368) itemoff 15683 itemsize 16
		range start 7225786368 end 7225802752 length 16384
	item 30 key (EXTENT_CSUM EXTENT_CSUM 7225802752) itemoff 15651 itemsize 32
		range start 7225802752 end 7225835520 length 32768
	item 31 key (EXTENT_CSUM EXTENT_CSUM 7225835520) itemoff 15615 itemsize 36
		range start 7225835520 end 7225872384 length 36864
	item 32 key (EXTENT_CSUM EXTENT_CSUM 7225872384) itemoff 15591 itemsize 24
		range start 7225872384 end 7225896960 length 24576
	item 33 key (EXTENT_CSUM EXTENT_CSUM 7225896960) itemoff 15555 itemsize 36
		range start 7225896960 end 7225933824 length 36864
	item 34 key (EXTENT_CSUM EXTENT_CSUM 7225933824) itemoff 15547 itemsize 8
		range start 7225933824 end 7225942016 length 8192
	item 35 key (EXTENT_CSUM EXTENT_CSUM 7225942016) itemoff 15531 itemsize 16
		range start 7225942016 end 7225958400 length 16384
	item 36 key (EXTENT_CSUM EXTENT_CSUM 7225958400) itemoff 15515 itemsize 16
		range start 7225958400 end 7225974784 length 16384
	item 37 key (EXTENT_CSUM EXTENT_CSUM 7225974784) itemoff 15503 itemsize 12
		range start 7225974784 end 7225987072 length 12288
	item 38 key (EXTENT_CSUM EXTENT_CSUM 7225995264) itemoff 15499 itemsize 4
		range start 7225995264 end 7225999360 length 4096
	item 39 key (EXTENT_CSUM EXTENT_CSUM 7225999360) itemoff 15491 itemsize 8
		range start 7225999360 end 7226007552 length 8192
	item 40 key (EXTENT_CSUM EXTENT_CSUM 7226007552) itemoff 15483 itemsize 8
		range start 7226007552 end 7226015744 length 8192
	item 41 key (EXTENT_CSUM EXTENT_CSUM 7226015744) itemoff 15467 itemsize 16
		range start 7226015744 end 7226032128 length 16384
	item 42 key (EXTENT_CSUM EXTENT_CSUM 7226032128) itemoff 15371 itemsize 96
		range start 7226032128 end 7226130432 length 98304
	item 43 key (EXTENT_CSUM EXTENT_CSUM 7226130432) itemoff 15347 itemsize 24
		range start 7226130432 end 7226155008 length 24576
	item 44 key (EXTENT_CSUM EXTENT_CSUM 7226155008) itemoff 15327 itemsize 20
		range start 7226155008 end 7226175488 length 20480
	item 45 key (EXTENT_CSUM EXTENT_CSUM 7226175488) itemoff 15319 itemsize 8
		range start 7226175488 end 7226183680 length 8192
	item 46 key (EXTENT_CSUM EXTENT_CSUM 7226183680) itemoff 15315 itemsize 4
		range start 7226183680 end 7226187776 length 4096
	item 47 key (EXTENT_CSUM EXTENT_CSUM 7226187776) itemoff 15295 itemsize 20
		range start 7226187776 end 7226208256 length 20480
	item 48 key (EXTENT_CSUM EXTENT_CSUM 7226208256) itemoff 15223 itemsize 72
		range start 7226208256 end 7226281984 length 73728
	item 49 key (EXTENT_CSUM EXTENT_CSUM 7226286080) itemoff 15207 itemsize 16
		range start 7226286080 end 7226302464 length 16384
	item 50 key (EXTENT_CSUM EXTENT_CSUM 7226302464) itemoff 15203 itemsize 4
		range start 7226302464 end 7226306560 length 4096
	item 51 key (EXTENT_CSUM EXTENT_CSUM 7226306560) itemoff 15195 itemsize 8
		range start 7226306560 end 7226314752 length 8192
	item 52 key (EXTENT_CSUM EXTENT_CSUM 7226314752) itemoff 15171 itemsize 24
		range start 7226314752 end 7226339328 length 24576
	item 53 key (EXTENT_CSUM EXTENT_CSUM 7226339328) itemoff 15139 itemsize 32
		range start 7226339328 end 7226372096 length 32768
	item 54 key (EXTENT_CSUM EXTENT_CSUM 7226372096) itemoff 15131 itemsize 8
		range start 7226372096 end 7226380288 length 8192
	item 55 key (EXTENT_CSUM EXTENT_CSUM 7226380288) itemoff 15111 itemsize 20
		range start 7226380288 end 7226400768 length 20480
	item 56 key (EXTENT_CSUM EXTENT_CSUM 7226400768) itemoff 15099 itemsize 12
		range start 7226400768 end 7226413056 length 12288
	item 57 key (EXTENT_CSUM EXTENT_CSUM 7226413056) itemoff 15075 itemsize 24
		range start 7226413056 end 7226437632 length 24576
	item 58 key (EXTENT_CSUM EXTENT_CSUM 7226437632) itemoff 15055 itemsize 20
		range start 7226437632 end 7226458112 length 20480
	item 59 key (EXTENT_CSUM EXTENT_CSUM 7226458112) itemoff 15047 itemsize 8
		range start 7226458112 end 7226466304 length 8192
	item 60 key (EXTENT_CSUM EXTENT_CSUM 7226466304) itemoff 15023 itemsize 24
		range start 7226466304 end 7226490880 length 24576
	item 61 key (EXTENT_CSUM EXTENT_CSUM 7226490880) itemoff 15007 itemsize 16
		range start 7226490880 end 7226507264 length 16384
	item 62 key (EXTENT_CSUM EXTENT_CSUM 7226507264) itemoff 14995 itemsize 12
		range start 7226507264 end 7226519552 length 12288
	item 63 key (EXTENT_CSUM EXTENT_CSUM 7226519552) itemoff 14987 itemsize 8
		range start 7226519552 end 7226527744 length 8192
	item 64 key (EXTENT_CSUM EXTENT_CSUM 7226527744) itemoff 14975 itemsize 12
		range start 7226527744 end 7226540032 length 12288
	item 65 key (EXTENT_CSUM EXTENT_CSUM 7226540032) itemoff 14955 itemsize 20
		range start 7226540032 end 7226560512 length 20480
	item 66 key (EXTENT_CSUM EXTENT_CSUM 7226560512) itemoff 14951 itemsize 4
		range start 7226560512 end 7226564608 length 4096
	item 67 key (EXTENT_CSUM EXTENT_CSUM 7226564608) itemoff 14927 itemsize 24
		range start 7226564608 end 7226589184 length 24576
	item 68 key (EXTENT_CSUM EXTENT_CSUM 7226589184) itemoff 14903 itemsize 24
		range start 7226589184 end 7226613760 length 24576
	item 69 key (EXTENT_CSUM EXTENT_CSUM 7226613760) itemoff 14891 itemsize 12
		range start 7226613760 end 7226626048 length 12288
	item 70 key (EXTENT_CSUM EXTENT_CSUM 7226626048) itemoff 14883 itemsize 8
		range start 7226626048 end 7226634240 length 8192
	item 71 key (EXTENT_CSUM EXTENT_CSUM 7226634240) itemoff 14859 itemsize 24
		range start 7226634240 end 7226658816 length 24576
	item 72 key (EXTENT_CSUM EXTENT_CSUM 7226658816) itemoff 14835 itemsize 24
		range start 7226658816 end 7226683392 length 24576
	item 73 key (EXTENT_CSUM EXTENT_CSUM 7226683392) itemoff 14803 itemsize 32
		range start 7226683392 end 7226716160 length 32768
	item 74 key (EXTENT_CSUM EXTENT_CSUM 7226716160) itemoff 14791 itemsize 12
		range start 7226716160 end 7226728448 length 12288
	item 75 key (EXTENT_CSUM EXTENT_CSUM 7226728448) itemoff 14779 itemsize 12
		range start 7226728448 end 7226740736 length 12288
	item 76 key (EXTENT_CSUM EXTENT_CSUM 7226740736) itemoff 14739 itemsize 40
		range start 7226740736 end 7226781696 length 40960
	item 77 key (EXTENT_CSUM EXTENT_CSUM 7226781696) itemoff 14703 itemsize 36
		range start 7226781696 end 7226818560 length 36864
	item 78 key (EXTENT_CSUM EXTENT_CSUM 7226818560) itemoff 14691 itemsize 12
		range start 7226818560 end 7226830848 length 12288
	item 79 key (EXTENT_CSUM EXTENT_CSUM 7226830848) itemoff 14683 itemsize 8
		range start 7226830848 end 7226839040 length 8192
	item 80 key (EXTENT_CSUM EXTENT_CSUM 7226839040) itemoff 14635 itemsize 48
		range start 7226839040 end 7226888192 length 49152
	item 81 key (EXTENT_CSUM EXTENT_CSUM 7226888192) itemoff 14583 itemsize 52
		range start 7226888192 end 7226941440 length 53248
	item 82 key (EXTENT_CSUM EXTENT_CSUM 7226941440) itemoff 14551 itemsize 32
		range start 7226941440 end 7226974208 length 32768
	item 83 key (EXTENT_CSUM EXTENT_CSUM 7226974208) itemoff 14539 itemsize 12
		range start 7226974208 end 7226986496 length 12288
	item 84 key (EXTENT_CSUM EXTENT_CSUM 7226986496) itemoff 14527 itemsize 12
		range start 7226986496 end 7226998784 length 12288
	item 85 key (EXTENT_CSUM EXTENT_CSUM 7226998784) itemoff 14503 itemsize 24
		range start 7226998784 end 7227023360 length 24576
	item 86 key (EXTENT_CSUM EXTENT_CSUM 7227023360) itemoff 14487 itemsize 16
		range start 7227023360 end 7227039744 length 16384
	item 87 key (EXTENT_CSUM EXTENT_CSUM 7227039744) itemoff 14475 itemsize 12
		range start 7227039744 end 7227052032 length 12288
	item 88 key (EXTENT_CSUM EXTENT_CSUM 7227052032) itemoff 14439 itemsize 36
		range start 7227052032 end 7227088896 length 36864
	item 89 key (EXTENT_CSUM EXTENT_CSUM 7227088896) itemoff 14423 itemsize 16
		range start 7227088896 end 7227105280 length 16384
	item 90 key (EXTENT_CSUM EXTENT_CSUM 7227105280) itemoff 14391 itemsize 32
		range start 7227105280 end 7227138048 length 32768
	item 91 key (EXTENT_CSUM EXTENT_CSUM 7227138048) itemoff 14355 itemsize 36
		range start 7227138048 end 7227174912 length 36864
	item 92 key (EXTENT_CSUM EXTENT_CSUM 7227174912) itemoff 14347 itemsize 8
		range start 7227174912 end 7227183104 length 8192
	item 93 key (EXTENT_CSUM EXTENT_CSUM 7227183104) itemoff 14331 itemsize 16
		range start 7227183104 end 7227199488 length 16384
	item 94 key (EXTENT_CSUM EXTENT_CSUM 7227199488) itemoff 14275 itemsize 56
		range start 7227199488 end 7227256832 length 57344
	item 95 key (EXTENT_CSUM EXTENT_CSUM 7227256832) itemoff 14255 itemsize 20
		range start 7227256832 end 7227277312 length 20480
	item 96 key (EXTENT_CSUM EXTENT_CSUM 7227277312) itemoff 14215 itemsize 40
		range start 7227277312 end 7227318272 length 40960
	item 97 key (EXTENT_CSUM EXTENT_CSUM 7227318272) itemoff 14183 itemsize 32
		range start 7227318272 end 7227351040 length 32768
	item 98 key (EXTENT_CSUM EXTENT_CSUM 7227351040) itemoff 14163 itemsize 20
		range start 7227351040 end 7227371520 length 20480
	item 99 key (EXTENT_CSUM EXTENT_CSUM 7227371520) itemoff 14127 itemsize 36
		range start 7227371520 end 7227408384 length 36864
	item 100 key (EXTENT_CSUM EXTENT_CSUM 7227408384) itemoff 14099 itemsize 28
		range start 7227408384 end 7227437056 length 28672
	item 101 key (EXTENT_CSUM EXTENT_CSUM 7227437056) itemoff 14087 itemsize 12
		range start 7227437056 end 7227449344 length 12288
	item 102 key (EXTENT_CSUM EXTENT_CSUM 7227449344) itemoff 14083 itemsize 4
		range start 7227449344 end 7227453440 length 4096
	item 103 key (EXTENT_CSUM EXTENT_CSUM 7227453440) itemoff 14075 itemsize 8
		range start 7227453440 end 7227461632 length 8192
	item 104 key (EXTENT_CSUM EXTENT_CSUM 7227461632) itemoff 14063 itemsize 12
		range start 7227461632 end 7227473920 length 12288
	item 105 key (EXTENT_CSUM EXTENT_CSUM 7227473920) itemoff 14035 itemsize 28
		range start 7227473920 end 7227502592 length 28672
	item 106 key (EXTENT_CSUM EXTENT_CSUM 7227502592) itemoff 14027 itemsize 8
		range start 7227502592 end 7227510784 length 8192
	item 107 key (EXTENT_CSUM EXTENT_CSUM 7227510784) itemoff 13975 itemsize 52
		range start 7227510784 end 7227564032 length 53248
	item 108 key (EXTENT_CSUM EXTENT_CSUM 7227564032) itemoff 13971 itemsize 4
		range start 7227564032 end 7227568128 length 4096
	item 109 key (EXTENT_CSUM EXTENT_CSUM 7227568128) itemoff 13943 itemsize 28
		range start 7227568128 end 7227596800 length 28672
	item 110 key (EXTENT_CSUM EXTENT_CSUM 7227596800) itemoff 13939 itemsize 4
		range start 7227596800 end 7227600896 length 4096
	item 111 key (EXTENT_CSUM EXTENT_CSUM 7227600896) itemoff 13927 itemsize 12
		range start 7227600896 end 7227613184 length 12288
	item 112 key (EXTENT_CSUM EXTENT_CSUM 7227617280) itemoff 13923 itemsize 4
		range start 7227617280 end 7227621376 length 4096
	item 113 key (EXTENT_CSUM EXTENT_CSUM 7227621376) itemoff 13887 itemsize 36
		range start 7227621376 end 7227658240 length 36864
	item 114 key (EXTENT_CSUM EXTENT_CSUM 7227658240) itemoff 13831 itemsize 56
		range start 7227658240 end 7227715584 length 57344
	item 115 key (EXTENT_CSUM EXTENT_CSUM 7227715584) itemoff 13819 itemsize 12
		range start 7227715584 end 7227727872 length 12288
	item 116 key (EXTENT_CSUM EXTENT_CSUM 7227727872) itemoff 13799 itemsize 20
		range start 7227727872 end 7227748352 length 20480
	item 117 key (EXTENT_CSUM EXTENT_CSUM 7227748352) itemoff 13759 itemsize 40
		range start 7227748352 end 7227789312 length 40960
	item 118 key (EXTENT_CSUM EXTENT_CSUM 7227789312) itemoff 13735 itemsize 24
		range start 7227789312 end 7227813888 length 24576
	item 119 key (EXTENT_CSUM EXTENT_CSUM 7227813888) itemoff 13715 itemsize 20
		range start 7227813888 end 7227834368 length 20480
	item 120 key (EXTENT_CSUM EXTENT_CSUM 7227834368) itemoff 13711 itemsize 4
		range start 7227834368 end 7227838464 length 4096
	item 121 key (EXTENT_CSUM EXTENT_CSUM 7227838464) itemoff 13691 itemsize 20
		range start 7227838464 end 7227858944 length 20480
	item 122 key (EXTENT_CSUM EXTENT_CSUM 7227858944) itemoff 13687 itemsize 4
		range start 7227858944 end 7227863040 length 4096
	item 123 key (EXTENT_CSUM EXTENT_CSUM 7227863040) itemoff 13655 itemsize 32
		range start 7227863040 end 7227895808 length 32768
	item 124 key (EXTENT_CSUM EXTENT_CSUM 7227895808) itemoff 13651 itemsize 4
		range start 7227895808 end 7227899904 length 4096
	item 125 key (EXTENT_CSUM EXTENT_CSUM 7227899904) itemoff 13599 itemsize 52
		range start 7227899904 end 7227953152 length 53248
	item 126 key (EXTENT_CSUM EXTENT_CSUM 7227953152) itemoff 13583 itemsize 16
		range start 7227953152 end 7227969536 length 16384
	item 127 key (EXTENT_CSUM EXTENT_CSUM 7227969536) itemoff 13579 itemsize 4
		range start 7227969536 end 7227973632 length 4096
	item 128 key (EXTENT_CSUM EXTENT_CSUM 7227973632) itemoff 13539 itemsize 40
		range start 7227973632 end 7228014592 length 40960
	item 129 key (EXTENT_CSUM EXTENT_CSUM 7228014592) itemoff 13503 itemsize 36
		range start 7228014592 end 7228051456 length 36864
	item 130 key (EXTENT_CSUM EXTENT_CSUM 7228051456) itemoff 13499 itemsize 4
		range start 7228051456 end 7228055552 length 4096
	item 131 key (EXTENT_CSUM EXTENT_CSUM 7228055552) itemoff 13459 itemsize 40
		range start 7228055552 end 7228096512 length 40960
	item 132 key (EXTENT_CSUM EXTENT_CSUM 7228096512) itemoff 13455 itemsize 4
		range start 7228096512 end 7228100608 length 4096
	item 133 key (EXTENT_CSUM EXTENT_CSUM 7228100608) itemoff 13431 itemsize 24
		range start 7228100608 end 7228125184 length 24576
	item 134 key (EXTENT_CSUM EXTENT_CSUM 7228125184) itemoff 13423 itemsize 8
		range start 7228125184 end 7228133376 length 8192
	item 135 key (EXTENT_CSUM EXTENT_CSUM 7228133376) itemoff 13419 itemsize 4
		range start 7228133376 end 7228137472 length 4096
	item 136 key (EXTENT_CSUM EXTENT_CSUM 7228137472) itemoff 13395 itemsize 24
		range start 7228137472 end 7228162048 length 24576
	item 137 key (EXTENT_CSUM EXTENT_CSUM 7228162048) itemoff 13383 itemsize 12
		range start 7228162048 end 7228174336 length 12288
	item 138 key (EXTENT_CSUM EXTENT_CSUM 7228174336) itemoff 13375 itemsize 8
		range start 7228174336 end 7228182528 length 8192
	item 139 key (EXTENT_CSUM EXTENT_CSUM 7228182528) itemoff 13359 itemsize 16
		range start 7228182528 end 7228198912 length 16384
	item 140 key (EXTENT_CSUM EXTENT_CSUM 7228198912) itemoff 13355 itemsize 4
		range start 7228198912 end 7228203008 length 4096
	item 141 key (EXTENT_CSUM EXTENT_CSUM 7228203008) itemoff 13327 itemsize 28
		range start 7228203008 end 7228231680 length 28672
	item 142 key (EXTENT_CSUM EXTENT_CSUM 7228231680) itemoff 13303 itemsize 24
		range start 7228231680 end 7228256256 length 24576
	item 143 key (EXTENT_CSUM EXTENT_CSUM 7228256256) itemoff 13295 itemsize 8
		range start 7228256256 end 7228264448 length 8192
	item 144 key (EXTENT_CSUM EXTENT_CSUM 7228264448) itemoff 13271 itemsize 24
		range start 7228264448 end 7228289024 length 24576
	item 145 key (EXTENT_CSUM EXTENT_CSUM 7228289024) itemoff 13195 itemsize 76
		range start 7228289024 end 7228366848 length 77824
	item 146 key (EXTENT_CSUM EXTENT_CSUM 7228366848) itemoff 13155 itemsize 40
		range start 7228366848 end 7228407808 length 40960
	item 147 key (EXTENT_CSUM EXTENT_CSUM 7228407808) itemoff 13135 itemsize 20
		range start 7228407808 end 7228428288 length 20480
	item 148 key (EXTENT_CSUM EXTENT_CSUM 7228428288) itemoff 13115 itemsize 20
		range start 7228428288 end 7228448768 length 20480
	item 149 key (EXTENT_CSUM EXTENT_CSUM 7228448768) itemoff 13095 itemsize 20
		range start 7228448768 end 7228469248 length 20480
	item 150 key (EXTENT_CSUM EXTENT_CSUM 7228469248) itemoff 13067 itemsize 28
		range start 7228469248 end 7228497920 length 28672
	item 151 key (EXTENT_CSUM EXTENT_CSUM 7228497920) itemoff 13063 itemsize 4
		range start 7228497920 end 7228502016 length 4096
	item 152 key (EXTENT_CSUM EXTENT_CSUM 7228502016) itemoff 13043 itemsize 20
		range start 7228502016 end 7228522496 length 20480
	item 153 key (EXTENT_CSUM EXTENT_CSUM 7228522496) itemoff 12999 itemsize 44
		range start 7228522496 end 7228567552 length 45056
	item 154 key (EXTENT_CSUM EXTENT_CSUM 7228567552) itemoff 12991 itemsize 8
		range start 7228567552 end 7228575744 length 8192
	item 155 key (EXTENT_CSUM EXTENT_CSUM 7228575744) itemoff 12967 itemsize 24
		range start 7228575744 end 7228600320 length 24576
	item 156 key (EXTENT_CSUM EXTENT_CSUM 7228600320) itemoff 12959 itemsize 8
		range start 7228600320 end 7228608512 length 8192
	item 157 key (EXTENT_CSUM EXTENT_CSUM 7228608512) itemoff 12951 itemsize 8
		range start 7228608512 end 7228616704 length 8192
	item 158 key (EXTENT_CSUM EXTENT_CSUM 7228616704) itemoff 12947 itemsize 4
		range start 7228616704 end 7228620800 length 4096
	item 159 key (EXTENT_CSUM EXTENT_CSUM 7228620800) itemoff 12927 itemsize 20
		range start 7228620800 end 7228641280 length 20480
	item 160 key (EXTENT_CSUM EXTENT_CSUM 7228641280) itemoff 12915 itemsize 12
		range start 7228641280 end 7228653568 length 12288
	item 161 key (EXTENT_CSUM EXTENT_CSUM 7228653568) itemoff 12911 itemsize 4
		range start 7228653568 end 7228657664 length 4096
	item 162 key (EXTENT_CSUM EXTENT_CSUM 7228657664) itemoff 12871 itemsize 40
		range start 7228657664 end 7228698624 length 40960
	item 163 key (EXTENT_CSUM EXTENT_CSUM 7228698624) itemoff 12863 itemsize 8
		range start 7228698624 end 7228706816 length 8192
	item 164 key (EXTENT_CSUM EXTENT_CSUM 7228706816) itemoff 12827 itemsize 36
		range start 7228706816 end 7228743680 length 36864
	item 165 key (EXTENT_CSUM EXTENT_CSUM 7228743680) itemoff 12819 itemsize 8
		range start 7228743680 end 7228751872 length 8192
	item 166 key (EXTENT_CSUM EXTENT_CSUM 7228751872) itemoff 12811 itemsize 8
		range start 7228751872 end 7228760064 length 8192
	item 167 key (EXTENT_CSUM EXTENT_CSUM 7228760064) itemoff 12799 itemsize 12
		range start 7228760064 end 7228772352 length 12288
	item 168 key (EXTENT_CSUM EXTENT_CSUM 7228772352) itemoff 12783 itemsize 16
		range start 7228772352 end 7228788736 length 16384
	item 169 key (EXTENT_CSUM EXTENT_CSUM 7228788736) itemoff 12779 itemsize 4
		range start 7228788736 end 7228792832 length 4096
	item 170 key (EXTENT_CSUM EXTENT_CSUM 7228792832) itemoff 12755 itemsize 24
		range start 7228792832 end 7228817408 length 24576
	item 171 key (EXTENT_CSUM EXTENT_CSUM 7228817408) itemoff 12743 itemsize 12
		range start 7228817408 end 7228829696 length 12288
	item 172 key (EXTENT_CSUM EXTENT_CSUM 7228829696) itemoff 12707 itemsize 36
		range start 7228829696 end 7228866560 length 36864
	item 173 key (EXTENT_CSUM EXTENT_CSUM 7228866560) itemoff 12687 itemsize 20
		range start 7228866560 end 7228887040 length 20480
	item 174 key (EXTENT_CSUM EXTENT_CSUM 7228887040) itemoff 12667 itemsize 20
		range start 7228887040 end 7228907520 length 20480
	item 175 key (EXTENT_CSUM EXTENT_CSUM 7228907520) itemoff 12655 itemsize 12
		range start 7228907520 end 7228919808 length 12288
	item 176 key (EXTENT_CSUM EXTENT_CSUM 7228919808) itemoff 12651 itemsize 4
		range start 7228919808 end 7228923904 length 4096
	item 177 key (EXTENT_CSUM EXTENT_CSUM 7228923904) itemoff 12647 itemsize 4
		range start 7228923904 end 7228928000 length 4096
	item 178 key (EXTENT_CSUM EXTENT_CSUM 7228928000) itemoff 12635 itemsize 12
		range start 7228928000 end 7228940288 length 12288
	item 179 key (EXTENT_CSUM EXTENT_CSUM 7228940288) itemoff 12631 itemsize 4
		range start 7228940288 end 7228944384 length 4096
	item 180 key (EXTENT_CSUM EXTENT_CSUM 7228944384) itemoff 12623 itemsize 8
		range start 7228944384 end 7228952576 length 8192
	item 181 key (EXTENT_CSUM EXTENT_CSUM 7228952576) itemoff 12599 itemsize 24
		range start 7228952576 end 7228977152 length 24576
	item 182 key (EXTENT_CSUM EXTENT_CSUM 7228977152) itemoff 12579 itemsize 20
		range start 7228977152 end 7228997632 length 20480
	item 183 key (EXTENT_CSUM EXTENT_CSUM 7228997632) itemoff 12559 itemsize 20
		range start 7228997632 end 7229018112 length 20480
	item 184 key (EXTENT_CSUM EXTENT_CSUM 7229018112) itemoff 12555 itemsize 4
		range start 7229018112 end 7229022208 length 4096
	item 185 key (EXTENT_CSUM EXTENT_CSUM 7229026304) itemoff 12551 itemsize 4
		range start 7229026304 end 7229030400 length 4096
	item 186 key (EXTENT_CSUM EXTENT_CSUM 7229030400) itemoff 12539 itemsize 12
		range start 7229030400 end 7229042688 length 12288
	item 187 key (EXTENT_CSUM EXTENT_CSUM 7229042688) itemoff 12527 itemsize 12
		range start 7229042688 end 7229054976 length 12288
	item 188 key (EXTENT_CSUM EXTENT_CSUM 7229054976) itemoff 12523 itemsize 4
		range start 7229054976 end 7229059072 length 4096
	item 189 key (EXTENT_CSUM EXTENT_CSUM 7229059072) itemoff 12499 itemsize 24
		range start 7229059072 end 7229083648 length 24576
	item 190 key (EXTENT_CSUM EXTENT_CSUM 7229083648) itemoff 12491 itemsize 8
		range start 7229083648 end 7229091840 length 8192
	item 191 key (EXTENT_CSUM EXTENT_CSUM 7229091840) itemoff 12483 itemsize 8
		range start 7229091840 end 7229100032 length 8192
	item 192 key (EXTENT_CSUM EXTENT_CSUM 7229100032) itemoff 12459 itemsize 24
		range start 7229100032 end 7229124608 length 24576
	item 193 key (EXTENT_CSUM EXTENT_CSUM 7229124608) itemoff 12443 itemsize 16
		range start 7229124608 end 7229140992 length 16384
	item 194 key (EXTENT_CSUM EXTENT_CSUM 7229140992) itemoff 12431 itemsize 12
		range start 7229140992 end 7229153280 length 12288
	item 195 key (EXTENT_CSUM EXTENT_CSUM 7229153280) itemoff 12427 itemsize 4
		range start 7229153280 end 7229157376 length 4096
	item 196 key (EXTENT_CSUM EXTENT_CSUM 7229157376) itemoff 12391 itemsize 36
		range start 7229157376 end 7229194240 length 36864
	item 197 key (EXTENT_CSUM EXTENT_CSUM 7229194240) itemoff 12375 itemsize 16
		range start 7229194240 end 7229210624 length 16384
	item 198 key (EXTENT_CSUM EXTENT_CSUM 7229210624) itemoff 12351 itemsize 24
		range start 7229210624 end 7229235200 length 24576
	item 199 key (EXTENT_CSUM EXTENT_CSUM 7229235200) itemoff 12319 itemsize 32
		range start 7229235200 end 7229267968 length 32768
	item 200 key (EXTENT_CSUM EXTENT_CSUM 7229267968) itemoff 12299 itemsize 20
		range start 7229267968 end 7229288448 length 20480
	item 201 key (EXTENT_CSUM EXTENT_CSUM 7229288448) itemoff 12283 itemsize 16
		range start 7229288448 end 7229304832 length 16384
	item 202 key (EXTENT_CSUM EXTENT_CSUM 7229304832) itemoff 12263 itemsize 20
		range start 7229304832 end 7229325312 length 20480
	item 203 key (EXTENT_CSUM EXTENT_CSUM 7229325312) itemoff 12251 itemsize 12
		range start 7229325312 end 7229337600 length 12288
	item 204 key (EXTENT_CSUM EXTENT_CSUM 7229337600) itemoff 12231 itemsize 20
		range start 7229337600 end 7229358080 length 20480
	item 205 key (EXTENT_CSUM EXTENT_CSUM 7229358080) itemoff 12223 itemsize 8
		range start 7229358080 end 7229366272 length 8192
	item 206 key (EXTENT_CSUM EXTENT_CSUM 7229366272) itemoff 12187 itemsize 36
		range start 7229366272 end 7229403136 length 36864
	item 207 key (EXTENT_CSUM EXTENT_CSUM 7229403136) itemoff 12171 itemsize 16
		range start 7229403136 end 7229419520 length 16384
	item 208 key (EXTENT_CSUM EXTENT_CSUM 7229419520) itemoff 12135 itemsize 36
		range start 7229419520 end 7229456384 length 36864
	item 209 key (EXTENT_CSUM EXTENT_CSUM 7229456384) itemoff 12103 itemsize 32
		range start 7229456384 end 7229489152 length 32768
	item 210 key (EXTENT_CSUM EXTENT_CSUM 7229489152) itemoff 12067 itemsize 36
		range start 7229489152 end 7229526016 length 36864
	item 211 key (EXTENT_CSUM EXTENT_CSUM 7229526016) itemoff 12027 itemsize 40
		range start 7229526016 end 7229566976 length 40960
	item 212 key (EXTENT_CSUM EXTENT_CSUM 7229566976) itemoff 12015 itemsize 12
		range start 7229566976 end 7229579264 length 12288
	item 213 key (EXTENT_CSUM EXTENT_CSUM 7229579264) itemoff 12011 itemsize 4
		range start 7229579264 end 7229583360 length 4096
	item 214 key (EXTENT_CSUM EXTENT_CSUM 7229583360) itemoff 11995 itemsize 16
		range start 7229583360 end 7229599744 length 16384
	item 215 key (EXTENT_CSUM EXTENT_CSUM 7229599744) itemoff 11971 itemsize 24
		range start 7229599744 end 7229624320 length 24576
	item 216 key (EXTENT_CSUM EXTENT_CSUM 7229624320) itemoff 11927 itemsize 44
		range start 7229624320 end 7229669376 length 45056
	item 217 key (EXTENT_CSUM EXTENT_CSUM 7229669376) itemoff 11923 itemsize 4
		range start 7229669376 end 7229673472 length 4096
	item 218 key (EXTENT_CSUM EXTENT_CSUM 7229673472) itemoff 11907 itemsize 16
		range start 7229673472 end 7229689856 length 16384
	item 219 key (EXTENT_CSUM EXTENT_CSUM 7229689856) itemoff 11903 itemsize 4
		range start 7229689856 end 7229693952 length 4096
	item 220 key (EXTENT_CSUM EXTENT_CSUM 7229693952) itemoff 11891 itemsize 12
		range start 7229693952 end 7229706240 length 12288
	item 221 key (EXTENT_CSUM EXTENT_CSUM 7229706240) itemoff 11883 itemsize 8
		range start 7229706240 end 7229714432 length 8192
	item 222 key (EXTENT_CSUM EXTENT_CSUM 7229714432) itemoff 11859 itemsize 24
		range start 7229714432 end 7229739008 length 24576
	item 223 key (EXTENT_CSUM EXTENT_CSUM 7229739008) itemoff 11835 itemsize 24
		range start 7229739008 end 7229763584 length 24576
	item 224 key (EXTENT_CSUM EXTENT_CSUM 7229763584) itemoff 11827 itemsize 8
		range start 7229763584 end 7229771776 length 8192
	item 225 key (EXTENT_CSUM EXTENT_CSUM 7229771776) itemoff 11815 itemsize 12
		range start 7229771776 end 7229784064 length 12288
	item 226 key (EXTENT_CSUM EXTENT_CSUM 7229784064) itemoff 11791 itemsize 24
		range start 7229784064 end 7229808640 length 24576
	item 227 key (EXTENT_CSUM EXTENT_CSUM 7229808640) itemoff 11771 itemsize 20
		range start 7229808640 end 7229829120 length 20480
	item 228 key (EXTENT_CSUM EXTENT_CSUM 7229829120) itemoff 11747 itemsize 24
		range start 7229829120 end 7229853696 length 24576
	item 229 key (EXTENT_CSUM EXTENT_CSUM 7229853696) itemoff 11727 itemsize 20
		range start 7229853696 end 7229874176 length 20480
	item 230 key (EXTENT_CSUM EXTENT_CSUM 7229874176) itemoff 11719 itemsize 8
		range start 7229874176 end 7229882368 length 8192
	item 231 key (EXTENT_CSUM EXTENT_CSUM 7229882368) itemoff 11707 itemsize 12
		range start 7229882368 end 7229894656 length 12288
	item 232 key (EXTENT_CSUM EXTENT_CSUM 7229894656) itemoff 11699 itemsize 8
		range start 7229894656 end 7229902848 length 8192
	item 233 key (EXTENT_CSUM EXTENT_CSUM 7229902848) itemoff 11691 itemsize 8
		range start 7229902848 end 7229911040 length 8192
	item 234 key (EXTENT_CSUM EXTENT_CSUM 7229911040) itemoff 11679 itemsize 12
		range start 7229911040 end 7229923328 length 12288
	item 235 key (EXTENT_CSUM EXTENT_CSUM 7229923328) itemoff 11663 itemsize 16
		range start 7229923328 end 7229939712 length 16384
	item 236 key (EXTENT_CSUM EXTENT_CSUM 7229939712) itemoff 11631 itemsize 32
		range start 7229939712 end 7229972480 length 32768
	item 237 key (EXTENT_CSUM EXTENT_CSUM 7229972480) itemoff 11627 itemsize 4
		range start 7229972480 end 7229976576 length 4096
	item 238 key (EXTENT_CSUM EXTENT_CSUM 7229976576) itemoff 11607 itemsize 20
		range start 7229976576 end 7229997056 length 20480
	item 239 key (EXTENT_CSUM EXTENT_CSUM 7229997056) itemoff 11595 itemsize 12
		range start 7229997056 end 7230009344 length 12288
	item 240 key (EXTENT_CSUM EXTENT_CSUM 7230009344) itemoff 11591 itemsize 4
		range start 7230009344 end 7230013440 length 4096
	item 241 key (EXTENT_CSUM EXTENT_CSUM 7230013440) itemoff 11555 itemsize 36
		range start 7230013440 end 7230050304 length 36864
	item 242 key (EXTENT_CSUM EXTENT_CSUM 7230050304) itemoff 11535 itemsize 20
		range start 7230050304 end 7230070784 length 20480
	item 243 key (EXTENT_CSUM EXTENT_CSUM 7230070784) itemoff 11531 itemsize 4
		range start 7230070784 end 7230074880 length 4096
	item 244 key (EXTENT_CSUM EXTENT_CSUM 7230074880) itemoff 11523 itemsize 8
		range start 7230074880 end 7230083072 length 8192
	item 245 key (EXTENT_CSUM EXTENT_CSUM 7230083072) itemoff 11467 itemsize 56
		range start 7230083072 end 7230140416 length 57344
	item 246 key (EXTENT_CSUM EXTENT_CSUM 7230140416) itemoff 11407 itemsize 60
		range start 7230140416 end 7230201856 length 61440
	item 247 key (EXTENT_CSUM EXTENT_CSUM 7230201856) itemoff 11371 itemsize 36
		range start 7230201856 end 7230238720 length 36864
	item 248 key (EXTENT_CSUM EXTENT_CSUM 7230238720) itemoff 11363 itemsize 8
		range start 7230238720 end 7230246912 length 8192
	item 249 key (EXTENT_CSUM EXTENT_CSUM 7230246912) itemoff 11355 itemsize 8
		range start 7230246912 end 7230255104 length 8192
	item 250 key (EXTENT_CSUM EXTENT_CSUM 7230255104) itemoff 11343 itemsize 12
		range start 7230255104 end 7230267392 length 12288
	item 251 key (EXTENT_CSUM EXTENT_CSUM 7230267392) itemoff 11263 itemsize 80
		range start 7230267392 end 7230349312 length 81920
	item 252 key (EXTENT_CSUM EXTENT_CSUM 7230349312) itemoff 11251 itemsize 12
		range start 7230349312 end 7230361600 length 12288
	item 253 key (EXTENT_CSUM EXTENT_CSUM 7230361600) itemoff 11247 itemsize 4
		range start 7230361600 end 7230365696 length 4096
	item 254 key (EXTENT_CSUM EXTENT_CSUM 7230365696) itemoff 11231 itemsize 16
		range start 7230365696 end 7230382080 length 16384
	item 255 key (EXTENT_CSUM EXTENT_CSUM 7230382080) itemoff 11195 itemsize 36
		range start 7230382080 end 7230418944 length 36864
	item 256 key (EXTENT_CSUM EXTENT_CSUM 7230418944) itemoff 11187 itemsize 8
		range start 7230418944 end 7230427136 length 8192
	item 257 key (EXTENT_CSUM EXTENT_CSUM 7230427136) itemoff 11183 itemsize 4
		range start 7230427136 end 7230431232 length 4096
	item 258 key (EXTENT_CSUM EXTENT_CSUM 7230431232) itemoff 11175 itemsize 8
		range start 7230431232 end 7230439424 length 8192
	item 259 key (EXTENT_CSUM EXTENT_CSUM 7230439424) itemoff 11163 itemsize 12
		range start 7230439424 end 7230451712 length 12288
	item 260 key (EXTENT_CSUM EXTENT_CSUM 7230451712) itemoff 11147 itemsize 16
		range start 7230451712 end 7230468096 length 16384
	item 261 key (EXTENT_CSUM EXTENT_CSUM 7230468096) itemoff 11127 itemsize 20
		range start 7230468096 end 7230488576 length 20480
	item 262 key (EXTENT_CSUM EXTENT_CSUM 7230488576) itemoff 11123 itemsize 4
		range start 7230488576 end 7230492672 length 4096
	item 263 key (EXTENT_CSUM EXTENT_CSUM 7230492672) itemoff 11115 itemsize 8
		range start 7230492672 end 7230500864 length 8192
	item 264 key (EXTENT_CSUM EXTENT_CSUM 7230500864) itemoff 11099 itemsize 16
		range start 7230500864 end 7230517248 length 16384
	item 265 key (EXTENT_CSUM EXTENT_CSUM 7230517248) itemoff 11063 itemsize 36
		range start 7230517248 end 7230554112 length 36864
	item 266 key (EXTENT_CSUM EXTENT_CSUM 7230554112) itemoff 11059 itemsize 4
		range start 7230554112 end 7230558208 length 4096
	item 267 key (EXTENT_CSUM EXTENT_CSUM 7230558208) itemoff 11055 itemsize 4
		range start 7230558208 end 7230562304 length 4096
	item 268 key (EXTENT_CSUM EXTENT_CSUM 7230562304) itemoff 11007 itemsize 48
		range start 7230562304 end 7230611456 length 49152
	item 269 key (EXTENT_CSUM EXTENT_CSUM 7230611456) itemoff 10987 itemsize 20
		range start 7230611456 end 7230631936 length 20480
	item 270 key (EXTENT_CSUM EXTENT_CSUM 7230631936) itemoff 10983 itemsize 4
		range start 7230631936 end 7230636032 length 4096
	item 271 key (EXTENT_CSUM EXTENT_CSUM 7230636032) itemoff 10967 itemsize 16
		range start 7230636032 end 7230652416 length 16384
	item 272 key (EXTENT_CSUM EXTENT_CSUM 7230652416) itemoff 10939 itemsize 28
		range start 7230652416 end 7230681088 length 28672
	item 273 key (EXTENT_CSUM EXTENT_CSUM 7230681088) itemoff 10927 itemsize 12
		range start 7230681088 end 7230693376 length 12288
	item 274 key (EXTENT_CSUM EXTENT_CSUM 7230693376) itemoff 10915 itemsize 12
		range start 7230693376 end 7230705664 length 12288
	item 275 key (EXTENT_CSUM EXTENT_CSUM 7230705664) itemoff 10911 itemsize 4
		range start 7230705664 end 7230709760 length 4096
	item 276 key (EXTENT_CSUM EXTENT_CSUM 7230709760) itemoff 10903 itemsize 8
		range start 7230709760 end 7230717952 length 8192
	item 277 key (EXTENT_CSUM EXTENT_CSUM 7230717952) itemoff 10863 itemsize 40
		range start 7230717952 end 7230758912 length 40960
	item 278 key (EXTENT_CSUM EXTENT_CSUM 7230758912) itemoff 10807 itemsize 56
		range start 7230758912 end 7230816256 length 57344
	item 279 key (EXTENT_CSUM EXTENT_CSUM 7230816256) itemoff 10803 itemsize 4
		range start 7230816256 end 7230820352 length 4096
	item 280 key (EXTENT_CSUM EXTENT_CSUM 7230820352) itemoff 10759 itemsize 44
		range start 7230820352 end 7230865408 length 45056
	item 281 key (EXTENT_CSUM EXTENT_CSUM 7230865408) itemoff 10739 itemsize 20
		range start 7230865408 end 7230885888 length 20480
	item 282 key (EXTENT_CSUM EXTENT_CSUM 7230885888) itemoff 10719 itemsize 20
		range start 7230885888 end 7230906368 length 20480
	item 283 key (EXTENT_CSUM EXTENT_CSUM 7230906368) itemoff 10703 itemsize 16
		range start 7230906368 end 7230922752 length 16384
	item 284 key (EXTENT_CSUM EXTENT_CSUM 7230922752) itemoff 10695 itemsize 8
		range start 7230922752 end 7230930944 length 8192
	item 285 key (EXTENT_CSUM EXTENT_CSUM 7230930944) itemoff 10683 itemsize 12
		range start 7230930944 end 7230943232 length 12288
	item 286 key (EXTENT_CSUM EXTENT_CSUM 7230943232) itemoff 10667 itemsize 16
		range start 7230943232 end 7230959616 length 16384
	item 287 key (EXTENT_CSUM EXTENT_CSUM 7230959616) itemoff 10635 itemsize 32
		range start 7230959616 end 7230992384 length 32768
	item 288 key (EXTENT_CSUM EXTENT_CSUM 7230992384) itemoff 10631 itemsize 4
		range start 7230992384 end 7230996480 length 4096
	item 289 key (EXTENT_CSUM EXTENT_CSUM 7230996480) itemoff 10607 itemsize 24
		range start 7230996480 end 7231021056 length 24576
	item 290 key (EXTENT_CSUM EXTENT_CSUM 7231021056) itemoff 10603 itemsize 4
		range start 7231021056 end 7231025152 length 4096
	item 291 key (EXTENT_CSUM EXTENT_CSUM 7231025152) itemoff 10551 itemsize 52
		range start 7231025152 end 7231078400 length 53248
	item 292 key (EXTENT_CSUM EXTENT_CSUM 7231078400) itemoff 10535 itemsize 16
		range start 7231078400 end 7231094784 length 16384
	item 293 key (EXTENT_CSUM EXTENT_CSUM 7231094784) itemoff 10523 itemsize 12
		range start 7231094784 end 7231107072 length 12288
	item 294 key (EXTENT_CSUM EXTENT_CSUM 7231107072) itemoff 10507 itemsize 16
		range start 7231107072 end 7231123456 length 16384
	item 295 key (EXTENT_CSUM EXTENT_CSUM 7231123456) itemoff 10479 itemsize 28
		range start 7231123456 end 7231152128 length 28672
	item 296 key (EXTENT_CSUM EXTENT_CSUM 7231152128) itemoff 10475 itemsize 4
		range start 7231152128 end 7231156224 length 4096
	item 297 key (EXTENT_CSUM EXTENT_CSUM 7231156224) itemoff 10459 itemsize 16
		range start 7231156224 end 7231172608 length 16384
	item 298 key (EXTENT_CSUM EXTENT_CSUM 7231172608) itemoff 10423 itemsize 36
		range start 7231172608 end 7231209472 length 36864
	item 299 key (EXTENT_CSUM EXTENT_CSUM 7231209472) itemoff 10407 itemsize 16
		range start 7231209472 end 7231225856 length 16384
	item 300 key (EXTENT_CSUM EXTENT_CSUM 7231225856) itemoff 10403 itemsize 4
		range start 7231225856 end 7231229952 length 4096
	item 301 key (EXTENT_CSUM EXTENT_CSUM 7231229952) itemoff 10395 itemsize 8
		range start 7231229952 end 7231238144 length 8192
	item 302 key (EXTENT_CSUM EXTENT_CSUM 7231238144) itemoff 10363 itemsize 32
		range start 7231238144 end 7231270912 length 32768
	item 303 key (EXTENT_CSUM EXTENT_CSUM 7231270912) itemoff 10299 itemsize 64
		range start 7231270912 end 7231336448 length 65536
	item 304 key (EXTENT_CSUM EXTENT_CSUM 7231336448) itemoff 10271 itemsize 28
		range start 7231336448 end 7231365120 length 28672
	item 305 key (EXTENT_CSUM EXTENT_CSUM 7231365120) itemoff 10255 itemsize 16
		range start 7231365120 end 7231381504 length 16384
	item 306 key (EXTENT_CSUM EXTENT_CSUM 7231381504) itemoff 10251 itemsize 4
		range start 7231381504 end 7231385600 length 4096
	item 307 key (EXTENT_CSUM EXTENT_CSUM 7231385600) itemoff 10227 itemsize 24
		range start 7231385600 end 7231410176 length 24576
	item 308 key (EXTENT_CSUM EXTENT_CSUM 7231410176) itemoff 10159 itemsize 68
		range start 7231410176 end 7231479808 length 69632
	item 309 key (EXTENT_CSUM EXTENT_CSUM 7231479808) itemoff 10143 itemsize 16
		range start 7231479808 end 7231496192 length 16384
	item 310 key (EXTENT_CSUM EXTENT_CSUM 7231496192) itemoff 10123 itemsize 20
		range start 7231496192 end 7231516672 length 20480
	item 311 key (EXTENT_CSUM EXTENT_CSUM 7231516672) itemoff 10115 itemsize 8
		range start 7231516672 end 7231524864 length 8192
	item 312 key (EXTENT_CSUM EXTENT_CSUM 7231524864) itemoff 10091 itemsize 24
		range start 7231524864 end 7231549440 length 24576
	item 313 key (EXTENT_CSUM EXTENT_CSUM 7231549440) itemoff 10079 itemsize 12
		range start 7231549440 end 7231561728 length 12288
	item 314 key (EXTENT_CSUM EXTENT_CSUM 7231561728) itemoff 10039 itemsize 40
		range start 7231561728 end 7231602688 length 40960
	item 315 key (EXTENT_CSUM EXTENT_CSUM 7231602688) itemoff 10007 itemsize 32
		range start 7231602688 end 7231635456 length 32768
	item 316 key (EXTENT_CSUM EXTENT_CSUM 7231635456) itemoff 10003 itemsize 4
		range start 7231635456 end 7231639552 length 4096
	item 317 key (EXTENT_CSUM EXTENT_CSUM 7231639552) itemoff 9987 itemsize 16
		range start 7231639552 end 7231655936 length 16384
	item 318 key (EXTENT_CSUM EXTENT_CSUM 7231655936) itemoff 9979 itemsize 8
		range start 7231655936 end 7231664128 length 8192
	item 319 key (EXTENT_CSUM EXTENT_CSUM 7231664128) itemoff 9963 itemsize 16
		range start 7231664128 end 7231680512 length 16384
	item 320 key (EXTENT_CSUM EXTENT_CSUM 7231680512) itemoff 9955 itemsize 8
		range start 7231680512 end 7231688704 length 8192
	item 321 key (EXTENT_CSUM EXTENT_CSUM 7231688704) itemoff 9951 itemsize 4
		range start 7231688704 end 7231692800 length 4096
	item 322 key (EXTENT_CSUM EXTENT_CSUM 7231692800) itemoff 9939 itemsize 12
		range start 7231692800 end 7231705088 length 12288
	item 323 key (EXTENT_CSUM EXTENT_CSUM 7231705088) itemoff 9931 itemsize 8
		range start 7231705088 end 7231713280 length 8192
	item 324 key (EXTENT_CSUM EXTENT_CSUM 7231713280) itemoff 9907 itemsize 24
		range start 7231713280 end 7231737856 length 24576
	item 325 key (EXTENT_CSUM EXTENT_CSUM 7231737856) itemoff 9895 itemsize 12
		range start 7231737856 end 7231750144 length 12288
	item 326 key (EXTENT_CSUM EXTENT_CSUM 7231750144) itemoff 9875 itemsize 20
		range start 7231750144 end 7231770624 length 20480
	item 327 key (EXTENT_CSUM EXTENT_CSUM 7231770624) itemoff 9871 itemsize 4
		range start 7231770624 end 7231774720 length 4096
	item 328 key (EXTENT_CSUM EXTENT_CSUM 7231774720) itemoff 9851 itemsize 20
		range start 7231774720 end 7231795200 length 20480
	item 329 key (EXTENT_CSUM EXTENT_CSUM 7231795200) itemoff 9823 itemsize 28
		range start 7231795200 end 7231823872 length 28672
	item 330 key (EXTENT_CSUM EXTENT_CSUM 7231823872) itemoff 9811 itemsize 12
		range start 7231823872 end 7231836160 length 12288
	item 331 key (EXTENT_CSUM EXTENT_CSUM 7231836160) itemoff 9807 itemsize 4
		range start 7231836160 end 7231840256 length 4096
	item 332 key (EXTENT_CSUM EXTENT_CSUM 7231840256) itemoff 9803 itemsize 4
		range start 7231840256 end 7231844352 length 4096
	item 333 key (EXTENT_CSUM EXTENT_CSUM 7231844352) itemoff 9783 itemsize 20
		range start 7231844352 end 7231864832 length 20480
	item 334 key (EXTENT_CSUM EXTENT_CSUM 7231864832) itemoff 9767 itemsize 16
		range start 7231864832 end 7231881216 length 16384
	item 335 key (EXTENT_CSUM EXTENT_CSUM 7231881216) itemoff 9731 itemsize 36
		range start 7231881216 end 7231918080 length 36864
	item 336 key (EXTENT_CSUM EXTENT_CSUM 7231918080) itemoff 9703 itemsize 28
		range start 7231918080 end 7231946752 length 28672
	item 337 key (EXTENT_CSUM EXTENT_CSUM 7231946752) itemoff 9671 itemsize 32
		range start 7231946752 end 7231979520 length 32768
	item 338 key (EXTENT_CSUM EXTENT_CSUM 7231979520) itemoff 9647 itemsize 24
		range start 7231979520 end 7232004096 length 24576
	item 339 key (EXTENT_CSUM EXTENT_CSUM 7232004096) itemoff 9643 itemsize 4
		range start 7232004096 end 7232008192 length 4096
	item 340 key (EXTENT_CSUM EXTENT_CSUM 7232008192) itemoff 9635 itemsize 8
		range start 7232008192 end 7232016384 length 8192
	item 341 key (EXTENT_CSUM EXTENT_CSUM 7232016384) itemoff 9575 itemsize 60
		range start 7232016384 end 7232077824 length 61440
	item 342 key (EXTENT_CSUM EXTENT_CSUM 7232077824) itemoff 9563 itemsize 12
		range start 7232077824 end 7232090112 length 12288
	item 343 key (EXTENT_CSUM EXTENT_CSUM 7232090112) itemoff 9551 itemsize 12
		range start 7232090112 end 7232102400 length 12288
	item 344 key (EXTENT_CSUM EXTENT_CSUM 7232102400) itemoff 9535 itemsize 16
		range start 7232102400 end 7232118784 length 16384
	item 345 key (EXTENT_CSUM EXTENT_CSUM 7232118784) itemoff 9527 itemsize 8
		range start 7232118784 end 7232126976 length 8192
	item 346 key (EXTENT_CSUM EXTENT_CSUM 7232126976) itemoff 9515 itemsize 12
		range start 7232126976 end 7232139264 length 12288
	item 347 key (EXTENT_CSUM EXTENT_CSUM 7232139264) itemoff 9511 itemsize 4
		range start 7232139264 end 7232143360 length 4096
	item 348 key (EXTENT_CSUM EXTENT_CSUM 7232143360) itemoff 9503 itemsize 8
		range start 7232143360 end 7232151552 length 8192
	item 349 key (EXTENT_CSUM EXTENT_CSUM 7232151552) itemoff 9499 itemsize 4
		range start 7232151552 end 7232155648 length 4096
	item 350 key (EXTENT_CSUM EXTENT_CSUM 7232155648) itemoff 9479 itemsize 20
		range start 7232155648 end 7232176128 length 20480
	item 351 key (EXTENT_CSUM EXTENT_CSUM 7232176128) itemoff 9427 itemsize 52
		range start 7232176128 end 7232229376 length 53248
	item 352 key (EXTENT_CSUM EXTENT_CSUM 7232229376) itemoff 9411 itemsize 16
		range start 7232229376 end 7232245760 length 16384
	item 353 key (EXTENT_CSUM EXTENT_CSUM 7232245760) itemoff 9347 itemsize 64
		range start 7232245760 end 7232311296 length 65536
	item 354 key (EXTENT_CSUM EXTENT_CSUM 7232319488) itemoff 9315 itemsize 32
		range start 7232319488 end 7232352256 length 32768
	item 355 key (EXTENT_CSUM EXTENT_CSUM 7232352256) itemoff 9299 itemsize 16
		range start 7232352256 end 7232368640 length 16384
	item 356 key (EXTENT_CSUM EXTENT_CSUM 7232368640) itemoff 9283 itemsize 16
		range start 7232368640 end 7232385024 length 16384
	item 357 key (EXTENT_CSUM EXTENT_CSUM 7232385024) itemoff 9279 itemsize 4
		range start 7232385024 end 7232389120 length 4096

btrfs-progs v6.15
leaf 202650877952 items 130 free space 6143 generation 460240 owner EXTENT_TREE leaf 202650877952 flags 0x1(WRITTEN) backref revision 1
fs uuid a6999502-9362-4fa7-b7b4-28095790217d
chunk uuid 710679da-fb76-4a51-9c84-8600a6e58c43

	item 0 key (16288821248 EXTENT_ITEM 12288) itemoff 16230 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df91b389398) extent data backref root 258 objectid 4884981 offset 83292160 count 1
	item 1 key (16288833536 EXTENT_ITEM 8192) itemoff 16177 itemsize 53
		refs 1 gen 388168 flags DATA
		(178 0x29740e4d562459f4) extent data backref root 257 objectid 23202041 offset 0 count 1
	item 2 key (16288841728 EXTENT_ITEM 16384) itemoff 16124 itemsize 53
		refs 1 gen 456882 flags DATA
		(178 0x29740e4d78342129) extent data backref root 257 objectid 28487889 offset 655360 count 1
	item 3 key (16288858112 EXTENT_ITEM 12288) itemoff 16071 itemsize 53
		refs 1 gen 429343 flags DATA
		(178 0x44d64df9e6c14b24) extent data backref root 258 objectid 31468215 offset 313806848 count 1
	item 4 key (16288870400 EXTENT_ITEM 20480) itemoff 16018 itemsize 53
		refs 1 gen 456822 flags DATA
		(178 0x29740e4d9fa16034) extent data backref root 257 objectid 28369018 offset 0 count 1
	item 5 key (16288890880 EXTENT_ITEM 24576) itemoff 15965 itemsize 53
		refs 1 gen 393336 flags DATA
		(178 0x44d64df9cee3171e) extent data backref root 258 objectid 28689233 offset 5111808 count 1
	item 6 key (16288915456 EXTENT_ITEM 8192) itemoff 15912 itemsize 53
		refs 1 gen 456827 flags DATA
		(178 0x29740e4d73c064b3) extent data backref root 257 objectid 28417426 offset 0 count 1
	item 7 key (16288923648 EXTENT_ITEM 36864) itemoff 15859 itemsize 53
		refs 1 gen 393178 flags DATA
		(178 0xdea30debcfbe2fe) extent data backref root 256 objectid 8009 offset 8519680 count 1
	item 8 key (16288985088 EXTENT_ITEM 12288) itemoff 15806 itemsize 53
		refs 2 gen 437733 flags DATA
		(178 0x44d64df97b4936c3) extent data backref root 258 objectid 4884981 offset 83496960 count 2
	item 9 key (16288997376 EXTENT_ITEM 8192) itemoff 15753 itemsize 53
		refs 1 gen 433260 flags DATA
		(178 0x44d64df9aadc9cfd) extent data backref root 258 objectid 33663587 offset 131072 count 1
	item 10 key (16289005568 EXTENT_ITEM 12288) itemoff 15700 itemsize 53
		refs 1 gen 433122 flags DATA
		(178 0x29740e4df5576fce) extent data backref root 257 objectid 26690967 offset 0 count 1
	item 11 key (16289017856 EXTENT_ITEM 16384) itemoff 15647 itemsize 53
		refs 1 gen 422653 flags DATA
		(178 0x44d64df9aa7253c2) extent data backref root 258 objectid 32265249 offset 4096 count 1
	item 12 key (16289034240 EXTENT_ITEM 8192) itemoff 15594 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df92f27412c) extent data backref root 258 objectid 4884981 offset 83533824 count 1
	item 13 key (16289042432 EXTENT_ITEM 4096) itemoff 15541 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df9bee7c7c8) extent data backref root 258 objectid 4884981 offset 119136256 count 1
	item 14 key (16289046528 EXTENT_ITEM 12288) itemoff 15488 itemsize 53
		refs 1 gen 434488 flags DATA
		(178 0x44d64df9fd9c683f) extent data backref root 258 objectid 33870485 offset 56885248 count 1
	item 15 key (16289058816 EXTENT_ITEM 12288) itemoff 15435 itemsize 53
		refs 1 gen 456827 flags DATA
		(178 0x29740e4d91cfc295) extent data backref root 257 objectid 28407840 offset 0 count 1
	item 16 key (16289071104 EXTENT_ITEM 28672) itemoff 15382 itemsize 53
		refs 1 gen 362448 flags DATA
		(178 0x44d64df95e9d6428) extent data backref root 258 objectid 24459707 offset 0 count 1
	item 17 key (16289099776 EXTENT_ITEM 24576) itemoff 15329 itemsize 53
		refs 1 gen 457483 flags DATA
		(178 0x44d64df957b90e93) extent data backref root 258 objectid 38263832 offset 0 count 1
	item 18 key (16289124352 EXTENT_ITEM 16384) itemoff 15276 itemsize 53
		refs 1 gen 460103 flags DATA
		(178 0x44d64df92a197e81) extent data backref root 258 objectid 38626509 offset 0 count 1
	item 19 key (16289140736 EXTENT_ITEM 8192) itemoff 15223 itemsize 53
		refs 1 gen 456827 flags DATA
		(178 0x29740e4d2fd9878c) extent data backref root 257 objectid 28417437 offset 0 count 1
	item 20 key (16289148928 EXTENT_ITEM 4096) itemoff 15170 itemsize 53
		refs 1 gen 441014 flags DATA
		(178 0x29740e4d0a980ef2) extent data backref root 257 objectid 27534151 offset 0 count 1
	item 21 key (16289153024 EXTENT_ITEM 8192) itemoff 15117 itemsize 53
		refs 1 gen 456821 flags DATA
		(178 0x29740e4dab1453ed) extent data backref root 257 objectid 28339107 offset 0 count 1
	item 22 key (16289161216 EXTENT_ITEM 12288) itemoff 15064 itemsize 53
		refs 1 gen 445384 flags DATA
		(178 0x44d64df933a982bb) extent data backref root 258 objectid 35619522 offset 786432 count 1
	item 23 key (16289173504 EXTENT_ITEM 40960) itemoff 15011 itemsize 53
		refs 1 gen 393193 flags DATA
		(178 0x29740e4d663bec2a) extent data backref root 257 objectid 22182907 offset 9830400 count 1
	item 24 key (16289214464 EXTENT_ITEM 24576) itemoff 14958 itemsize 53
		refs 1 gen 398688 flags DATA
		(178 0xdea30dec24563c5) extent data backref root 256 objectid 8131 offset 14532608 count 1
	item 25 key (16289239040 EXTENT_ITEM 16384) itemoff 14905 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df9d30d1ed1) extent data backref root 258 objectid 4884981 offset 83251200 count 1
	item 26 key (16289255424 EXTENT_ITEM 8192) itemoff 14852 itemsize 53
		refs 1 gen 422653 flags DATA
		(178 0x44d64df9df804e60) extent data backref root 258 objectid 32265441 offset 0 count 1
	item 27 key (16289263616 EXTENT_ITEM 12288) itemoff 14799 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df901b2525f) extent data backref root 258 objectid 4884981 offset 83673088 count 1
	item 28 key (16289275904 EXTENT_ITEM 4096) itemoff 14746 itemsize 53
		refs 1 gen 441014 flags DATA
		(178 0x29740e4d62894cd8) extent data backref root 257 objectid 27534157 offset 0 count 1
	item 29 key (16289280000 EXTENT_ITEM 24576) itemoff 14693 itemsize 53
		refs 1 gen 428867 flags DATA
		(178 0x29740e4dc482a782) extent data backref root 257 objectid 26234191 offset 9437184 count 1
	item 30 key (16289304576 EXTENT_ITEM 12288) itemoff 14640 itemsize 53
		refs 1 gen 456821 flags DATA
		(178 0x29740e4ddd075b7f) extent data backref root 257 objectid 28315939 offset 0 count 1
	item 31 key (16289316864 EXTENT_ITEM 24576) itemoff 14587 itemsize 53
		refs 1 gen 433941 flags DATA
		(178 0x44d64df9eab264c5) extent data backref root 258 objectid 33782732 offset 14811136 count 1
	item 32 key (16289341440 EXTENT_ITEM 36864) itemoff 14534 itemsize 53
		refs 1 gen 459405 flags DATA
		(178 0x44d64df9f509298a) extent data backref root 258 objectid 38533815 offset 0 count 1
	item 33 key (16289378304 EXTENT_ITEM 28672) itemoff 14481 itemsize 53
		refs 1 gen 379987 flags DATA
		(178 0x44d64df93d928712) extent data backref root 258 objectid 26538546 offset 1441792 count 1
	item 34 key (16289406976 EXTENT_ITEM 12288) itemoff 14428 itemsize 53
		refs 1 gen 434488 flags DATA
		(178 0x44d64df93297aa24) extent data backref root 258 objectid 33870485 offset 69337088 count 1
	item 35 key (16289419264 EXTENT_ITEM 12288) itemoff 14375 itemsize 53
		refs 1 gen 445384 flags DATA
		(178 0x44d64df9f5ad2e50) extent data backref root 258 objectid 35619526 offset 786432 count 1
	item 36 key (16289431552 EXTENT_ITEM 8192) itemoff 14322 itemsize 53
		refs 1 gen 433121 flags DATA
		(178 0x29740e4dd66db62e) extent data backref root 257 objectid 26648693 offset 131072 count 1
	item 37 key (16289439744 EXTENT_ITEM 4096) itemoff 14269 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df96cb7f1ce) extent data backref root 258 objectid 4884981 offset 119189504 count 1
	item 38 key (16289443840 EXTENT_ITEM 4096) itemoff 14216 itemsize 53
		refs 1 gen 456827 flags DATA
		(178 0x29740e4d8231aee5) extent data backref root 257 objectid 28452236 offset 0 count 1
	item 39 key (16289447936 EXTENT_ITEM 36864) itemoff 14163 itemsize 53
		refs 1 gen 406313 flags DATA
		(178 0xdea30de8fd95e8e) extent data backref root 256 objectid 8280 offset 20049920 count 1
	item 40 key (16289484800 EXTENT_ITEM 16384) itemoff 14110 itemsize 53
		refs 1 gen 445267 flags DATA
		(178 0x44d64df93f6f9bab) extent data backref root 258 objectid 35592582 offset 165281792 count 1
	item 41 key (16289501184 EXTENT_ITEM 45056) itemoff 14057 itemsize 53
		refs 1 gen 386194 flags DATA
		(178 0x44d64df99e30d852) extent data backref root 258 objectid 27436445 offset 0 count 1
	item 42 key (16289546240 EXTENT_ITEM 40960) itemoff 14004 itemsize 53
		refs 1 gen 459397 flags DATA
		(178 0x44d64df9467a14ea) extent data backref root 258 objectid 38445696 offset 1134592 count 1
	item 43 key (16289587200 EXTENT_ITEM 12288) itemoff 13951 itemsize 53
		refs 1 gen 393206 flags DATA
		(178 0x29740e4d58b972bd) extent data backref root 257 objectid 23547967 offset 0 count 1
	item 44 key (16289599488 EXTENT_ITEM 12288) itemoff 13898 itemsize 53
		refs 1 gen 434488 flags DATA
		(178 0x44d64df95bee3d90) extent data backref root 258 objectid 33870485 offset 69468160 count 1
	item 45 key (16289611776 EXTENT_ITEM 16384) itemoff 13845 itemsize 53
		refs 1 gen 393334 flags DATA
		(178 0x44d64df9cdf3b857) extent data backref root 258 objectid 28643204 offset 131072 count 1
	item 46 key (16289628160 EXTENT_ITEM 16384) itemoff 13792 itemsize 53
		refs 1 gen 393336 flags DATA
		(178 0x44d64df92e392f05) extent data backref root 258 objectid 28689226 offset 142868480 count 1
	item 47 key (16289644544 EXTENT_ITEM 8192) itemoff 13739 itemsize 53
		refs 1 gen 422653 flags DATA
		(178 0x44d64df9cc782e93) extent data backref root 258 objectid 32265442 offset 0 count 1
	item 48 key (16289652736 EXTENT_ITEM 24576) itemoff 13686 itemsize 53
		refs 1 gen 443262 flags DATA
		(178 0x44d64df9b6960e24) extent data backref root 258 objectid 35105385 offset 0 count 1
	item 49 key (16289677312 EXTENT_ITEM 8192) itemoff 13633 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df920889ff7) extent data backref root 258 objectid 4884981 offset 84160512 count 1
	item 50 key (16289685504 EXTENT_ITEM 4096) itemoff 13580 itemsize 53
		refs 1 gen 433260 flags DATA
		(178 0x44d64df99b0a75a4) extent data backref root 258 objectid 33663605 offset 131072 count 1
	item 51 key (16289689600 EXTENT_ITEM 16384) itemoff 13527 itemsize 53
		refs 1 gen 399422 flags DATA
		(178 0x44d64df96985e322) extent data backref root 258 objectid 29498955 offset 131072 count 1
	item 52 key (16289705984 EXTENT_ITEM 16384) itemoff 13474 itemsize 53
		refs 2 gen 437733 flags DATA
		(178 0x44d64df992467622) extent data backref root 258 objectid 4884981 offset 84033536 count 2
	item 53 key (16289722368 EXTENT_ITEM 53248) itemoff 13421 itemsize 53
		refs 1 gen 393178 flags DATA
		(178 0xdea30deeb0f2870) extent data backref root 256 objectid 8009 offset 7471104 count 1
	item 54 key (16289775616 EXTENT_ITEM 12288) itemoff 13368 itemsize 53
		refs 1 gen 456827 flags DATA
		(178 0x29740e4d07221588) extent data backref root 257 objectid 28407899 offset 4718592 count 1
	item 55 key (16289787904 EXTENT_ITEM 8192) itemoff 13315 itemsize 53
		refs 1 gen 433260 flags DATA
		(178 0x44d64df909006045) extent data backref root 258 objectid 33663604 offset 655360 count 1
	item 56 key (16289796096 EXTENT_ITEM 8192) itemoff 13262 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df90e51ef62) extent data backref root 258 objectid 4884981 offset 84197376 count 1
	item 57 key (16289804288 EXTENT_ITEM 8192) itemoff 13209 itemsize 53
		refs 1 gen 433260 flags DATA
		(178 0x44d64df97afe18a9) extent data backref root 258 objectid 33663607 offset 131072 count 1
	item 58 key (16289812480 EXTENT_ITEM 16384) itemoff 13156 itemsize 53
		refs 1 gen 416487 flags DATA
		(178 0x29740e4dd3c210dc) extent data backref root 257 objectid 25561126 offset 0 count 1
	item 59 key (16289828864 EXTENT_ITEM 61440) itemoff 13103 itemsize 53
		refs 1 gen 443061 flags DATA
		(178 0x29740e4d98475919) extent data backref root 257 objectid 27662910 offset 131072 count 1
	item 60 key (16289898496 EXTENT_ITEM 8192) itemoff 13050 itemsize 53
		refs 1 gen 460104 flags DATA
		(178 0x44d64df97ac7831b) extent data backref root 258 objectid 38627029 offset 0 count 1
	item 61 key (16289906688 EXTENT_ITEM 8192) itemoff 12997 itemsize 53
		refs 1 gen 460104 flags DATA
		(178 0x44d64df99b33ee16) extent data backref root 258 objectid 38627031 offset 0 count 1
	item 62 key (16289914880 EXTENT_ITEM 8192) itemoff 12944 itemsize 53
		refs 1 gen 460104 flags DATA
		(178 0x44d64df9c72a0d29) extent data backref root 258 objectid 38627032 offset 0 count 1
	item 63 key (16289923072 EXTENT_ITEM 20480) itemoff 12891 itemsize 53
		refs 1 gen 457483 flags DATA
		(178 0x44d64df9ccfb48c8) extent data backref root 258 objectid 38263887 offset 0 count 1
	item 64 key (16289943552 EXTENT_ITEM 8192) itemoff 12838 itemsize 53
		refs 1 gen 443101 flags DATA
		(178 0x29740e4ddef2f580) extent data backref root 257 objectid 27703844 offset 0 count 1
	item 65 key (16289951744 EXTENT_ITEM 12288) itemoff 12785 itemsize 53
		refs 1 gen 388167 flags DATA
		(178 0x29740e4dbf4ae5b7) extent data backref root 257 objectid 23186425 offset 0 count 1
	item 66 key (16289964032 EXTENT_ITEM 57344) itemoff 12732 itemsize 53
		refs 1 gen 443061 flags DATA
		(178 0x29740e4d19904415) extent data backref root 257 objectid 27662936 offset 917504 count 1
	item 67 key (16290021376 EXTENT_ITEM 4096) itemoff 12679 itemsize 53
		refs 1 gen 456827 flags DATA
		(178 0x29740e4d3a02689b) extent data backref root 257 objectid 28452242 offset 0 count 1
	item 68 key (16290025472 EXTENT_ITEM 32768) itemoff 12626 itemsize 53
		refs 1 gen 456783 flags DATA
		(178 0x44d64df95d278b9f) extent data backref root 258 objectid 38189779 offset 146407424 count 1
	item 69 key (16290058240 EXTENT_ITEM 4096) itemoff 12573 itemsize 53
		refs 1 gen 441014 flags DATA
		(178 0x29740e4d28b68758) extent data backref root 257 objectid 27534162 offset 0 count 1
	item 70 key (16290062336 EXTENT_ITEM 20480) itemoff 12520 itemsize 53
		refs 1 gen 393336 flags DATA
		(178 0x44d64df969897674) extent data backref root 258 objectid 28689247 offset 4194304 count 1
	item 71 key (16290082816 EXTENT_ITEM 12288) itemoff 12467 itemsize 53
		refs 1 gen 393206 flags DATA
		(178 0x29740e4da056e654) extent data backref root 257 objectid 23547972 offset 0 count 1
	item 72 key (16290095104 EXTENT_ITEM 8192) itemoff 12414 itemsize 53
		refs 1 gen 422653 flags DATA
		(178 0x44d64df9eb88ef75) extent data backref root 258 objectid 32265444 offset 0 count 1
	item 73 key (16290103296 EXTENT_ITEM 4096) itemoff 12361 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df9900569ff) extent data backref root 258 objectid 4884981 offset 119234560 count 1
	item 74 key (16290107392 EXTENT_ITEM 40960) itemoff 12308 itemsize 53
		refs 1 gen 416511 flags DATA
		(178 0x44d64df9bcfd893b) extent data backref root 258 objectid 31470669 offset 167378944 count 1
	item 75 key (16290148352 EXTENT_ITEM 20480) itemoff 12255 itemsize 53
		refs 1 gen 433123 flags DATA
		(178 0x29740e4dfd5aaf8b) extent data backref root 257 objectid 26797062 offset 131072 count 1
	item 76 key (16290168832 EXTENT_ITEM 8192) itemoff 12202 itemsize 53
		refs 1 gen 433260 flags DATA
		(178 0x44d64df9d4ebf668) extent data backref root 258 objectid 33663609 offset 131072 count 1
	item 77 key (16290177024 EXTENT_ITEM 40960) itemoff 12149 itemsize 53
		refs 1 gen 406309 flags DATA
		(178 0x44d64df9cd7ae46b) extent data backref root 258 objectid 30361388 offset 786432 count 1
	item 78 key (16290217984 EXTENT_ITEM 12288) itemoff 12096 itemsize 53
		refs 1 gen 393206 flags DATA
		(178 0x29740e4dfc4f056b) extent data backref root 257 objectid 23547979 offset 0 count 1
	item 79 key (16290230272 EXTENT_ITEM 8192) itemoff 12043 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df94061fd60) extent data backref root 258 objectid 4884981 offset 84275200 count 1
	item 80 key (16290238464 EXTENT_ITEM 4096) itemoff 11990 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df990490a19) extent data backref root 258 objectid 4884981 offset 119402496 count 1
	item 81 key (16290242560 EXTENT_ITEM 28672) itemoff 11937 itemsize 53
		refs 1 gen 397730 flags DATA
		(178 0x44d64df96be72e03) extent data backref root 258 objectid 29298174 offset 655360 count 1
	item 82 key (16290279424 EXTENT_ITEM 20480) itemoff 11884 itemsize 53
		refs 1 gen 382592 flags DATA
		(178 0x29740e4d1a6ff578) extent data backref root 257 objectid 22869370 offset 11403264 count 1
	item 83 key (16290299904 EXTENT_ITEM 24576) itemoff 11831 itemsize 53
		refs 1 gen 433121 flags DATA
		(178 0x29740e4d6e3b7fe0) extent data backref root 257 objectid 26637781 offset 131072 count 1
	item 84 key (16290328576 EXTENT_ITEM 4096) itemoff 11778 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df9cdc03591) extent data backref root 258 objectid 4884981 offset 119431168 count 1
	item 85 key (16290332672 EXTENT_ITEM 4096) itemoff 11725 itemsize 53
		refs 1 gen 393204 flags DATA
		(178 0x29740e4d97f1b74a) extent data backref root 257 objectid 169769 offset 14618624 count 1
	item 86 key (16290336768 EXTENT_ITEM 16384) itemoff 11672 itemsize 53
		refs 1 gen 375477 flags DATA
		(178 0x29740e4d63b46c55) extent data backref root 257 objectid 22458835 offset 0 count 1
	item 87 key (16290353152 EXTENT_ITEM 12288) itemoff 11619 itemsize 53
		refs 2 gen 437733 flags DATA
		(178 0x44d64df9bce8bbf3) extent data backref root 258 objectid 4884981 offset 84299776 count 2
	item 88 key (16290365440 EXTENT_ITEM 4096) itemoff 11566 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df999ae427e) extent data backref root 258 objectid 4884981 offset 119459840 count 1
	item 89 key (16290369536 EXTENT_ITEM 4096) itemoff 11513 itemsize 53
		refs 1 gen 453652 flags DATA
		(178 0x29740e4d63de9f9e) extent data backref root 257 objectid 28199966 offset 0 count 1
	item 90 key (16290373632 EXTENT_ITEM 12288) itemoff 11460 itemsize 53
		refs 1 gen 446169 flags DATA
		(178 0x44d64df94d567093) extent data backref root 258 objectid 35779356 offset 0 count 1
	item 91 key (16290385920 EXTENT_ITEM 28672) itemoff 11407 itemsize 53
		refs 1 gen 457148 flags DATA
		(178 0x44d64df9382a6ec8) extent data backref root 258 objectid 38212684 offset 29229056 count 1
	item 92 key (16290414592 EXTENT_ITEM 4096) itemoff 11354 itemsize 53
		refs 1 gen 443061 flags DATA
		(178 0x29740e4d42601298) extent data backref root 257 objectid 27673518 offset 0 count 1
	item 93 key (16290418688 EXTENT_ITEM 4096) itemoff 11301 itemsize 53
		refs 1 gen 393204 flags DATA
		(178 0x29740e4d36f1ce51) extent data backref root 257 objectid 169769 offset 14622720 count 1
	item 94 key (16290422784 EXTENT_ITEM 20480) itemoff 11248 itemsize 53
		refs 1 gen 445267 flags DATA
		(178 0x44d64df9bcdb343d) extent data backref root 258 objectid 35592582 offset 70516736 count 1
	item 95 key (16290443264 EXTENT_ITEM 49152) itemoff 11195 itemsize 53
		refs 1 gen 393178 flags DATA
		(178 0xdea30ded582754a) extent data backref root 256 objectid 8009 offset 8650752 count 1
	item 96 key (16290492416 EXTENT_ITEM 61440) itemoff 11142 itemsize 53
		refs 1 gen 406651 flags DATA
		(178 0x44d64df9618c95d0) extent data backref root 258 objectid 30429694 offset 0 count 1
	item 97 key (16290553856 EXTENT_ITEM 36864) itemoff 11089 itemsize 53
		refs 1 gen 452099 flags DATA
		(178 0x44d64df96179dec9) extent data backref root 258 objectid 37240349 offset 0 count 1
	item 98 key (16290590720 EXTENT_ITEM 8192) itemoff 11036 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df974dd36ba) extent data backref root 258 objectid 4884981 offset 84340736 count 1
	item 99 key (16290598912 EXTENT_ITEM 4096) itemoff 10983 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df9584759f2) extent data backref root 258 objectid 4884981 offset 119480320 count 1
	item 100 key (16290603008 EXTENT_ITEM 16384) itemoff 10930 itemsize 53
		refs 1 gen 392182 flags DATA
		(178 0x44d64df974917de3) extent data backref root 258 objectid 28415038 offset 0 count 1
	item 101 key (16290619392 EXTENT_ITEM 28672) itemoff 10877 itemsize 53
		refs 1 gen 452097 flags DATA
		(178 0x44d64df999ba3591) extent data backref root 258 objectid 37239240 offset 0 count 1
	item 102 key (16290648064 EXTENT_ITEM 20480) itemoff 10824 itemsize 53
		refs 1 gen 455523 flags DATA
		(178 0x44d64df940c55b51) extent data backref root 258 objectid 37980051 offset 0 count 1
	item 103 key (16290668544 EXTENT_ITEM 4096) itemoff 10771 itemsize 53
		refs 1 gen 453652 flags DATA
		(178 0x29740e4d91d29260) extent data backref root 257 objectid 28199967 offset 0 count 1
	item 104 key (16290672640 EXTENT_ITEM 49152) itemoff 10718 itemsize 53
		refs 1 gen 443061 flags DATA
		(178 0x29740e4d6216ccf7) extent data backref root 257 objectid 27662936 offset 1310720 count 1
	item 105 key (16290721792 EXTENT_ITEM 8192) itemoff 10665 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df9b5db57be) extent data backref root 258 objectid 4884981 offset 84852736 count 1
	item 106 key (16290729984 EXTENT_ITEM 36864) itemoff 10612 itemsize 53
		refs 1 gen 433260 flags DATA
		(178 0x44d64df9c77515a4) extent data backref root 258 objectid 33663567 offset 4325376 count 1
	item 107 key (16290766848 EXTENT_ITEM 4096) itemoff 10559 itemsize 53
		refs 1 gen 456827 flags DATA
		(178 0x29740e4dc80e6565) extent data backref root 257 objectid 28452243 offset 0 count 1
	item 108 key (16290770944 EXTENT_ITEM 20480) itemoff 10506 itemsize 53
		refs 1 gen 437264 flags DATA
		(178 0x44d64df9937c146d) extent data backref root 258 objectid 34274723 offset 4096 count 1
	item 109 key (16290791424 EXTENT_ITEM 8192) itemoff 10453 itemsize 53
		refs 1 gen 422653 flags DATA
		(178 0x44d64df91984e28b) extent data backref root 258 objectid 32265445 offset 0 count 1
	item 110 key (16290799616 EXTENT_ITEM 32768) itemoff 10400 itemsize 53
		refs 1 gen 436803 flags DATA
		(178 0x44d64df9be7c1f00) extent data backref root 258 objectid 12714580 offset 655360 count 1
	item 111 key (16290832384 EXTENT_ITEM 28672) itemoff 10347 itemsize 53
		refs 1 gen 457115 flags DATA
		(178 0x44d64df98f2d28ac) extent data backref root 258 objectid 38206097 offset 131072 count 1
	item 112 key (16290865152 EXTENT_ITEM 4096) itemoff 10294 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df90c292e1d) extent data backref root 258 objectid 4884981 offset 119508992 count 1
	item 113 key (16290869248 EXTENT_ITEM 16384) itemoff 10241 itemsize 53
		refs 1 gen 460103 flags DATA
		(178 0x44d64df9a840c13e) extent data backref root 258 objectid 38626531 offset 0 count 1
	item 114 key (16290885632 EXTENT_ITEM 8192) itemoff 10188 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df9061fef51) extent data backref root 258 objectid 4884981 offset 84967424 count 1
	item 115 key (16290893824 EXTENT_ITEM 12288) itemoff 10135 itemsize 53
		refs 1 gen 400043 flags DATA
		(178 0x29740e4d6200abdc) extent data backref root 257 objectid 24576267 offset 0 count 1
	item 116 key (16290906112 EXTENT_ITEM 24576) itemoff 10082 itemsize 53
		refs 1 gen 433941 flags DATA
		(178 0x44d64df983cbf371) extent data backref root 258 objectid 33782732 offset 14942208 count 1
	item 117 key (16290930688 EXTENT_ITEM 8192) itemoff 10029 itemsize 53
		refs 1 gen 402680 flags DATA
		(178 0x29740e4d7e5c6249) extent data backref root 257 objectid 24697659 offset 0 count 1
	item 118 key (16290938880 EXTENT_ITEM 40960) itemoff 9976 itemsize 53
		refs 1 gen 445266 flags DATA
		(178 0x44d64df9b0f6f92f) extent data backref root 258 objectid 35561230 offset 917504 count 1
	item 119 key (16290988032 EXTENT_ITEM 12288) itemoff 9923 itemsize 53
		refs 1 gen 394481 flags DATA
		(178 0x44d64df9ab71e1f6) extent data backref root 258 objectid 28826541 offset 0 count 1
	item 120 key (16291000320 EXTENT_ITEM 12288) itemoff 9870 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df914974d43) extent data backref root 258 objectid 4884981 offset 84434944 count 1
	item 121 key (16291012608 EXTENT_ITEM 53248) itemoff 9817 itemsize 53
		refs 1 gen 443061 flags DATA
		(178 0x29740e4dd0e3fb80) extent data backref root 257 objectid 27662936 offset 1048576 count 1
	item 122 key (16291065856 EXTENT_ITEM 8192) itemoff 9764 itemsize 53
		refs 1 gen 443101 flags DATA
		(178 0x29740e4d70e71b41) extent data backref root 257 objectid 27703850 offset 0 count 1
	item 123 key (16291074048 EXTENT_ITEM 32768) itemoff 9711 itemsize 53
		refs 1 gen 457148 flags DATA
		(178 0x44d64df93b1c2322) extent data backref root 258 objectid 38212696 offset 655360 count 1
	item 124 key (16291106816 EXTENT_ITEM 4096) itemoff 9658 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df967ab6155) extent data backref root 258 objectid 4884981 offset 119549952 count 1
	item 125 key (16291110912 EXTENT_ITEM 45056) itemoff 9605 itemsize 53
		refs 1 gen 445266 flags DATA
		(178 0x44d64df97745e207) extent data backref root 258 objectid 35561213 offset 393216 count 1
	item 126 key (16291155968 EXTENT_ITEM 36864) itemoff 9552 itemsize 53
		refs 1 gen 406313 flags DATA
		(178 0xdea30de5455fe4d) extent data backref root 256 objectid 8280 offset 20180992 count 1
	item 127 key (16291192832 EXTENT_ITEM 49152) itemoff 9499 itemsize 53
		refs 1 gen 455958 flags DATA
		(178 0x44d64df9598457bc) extent data backref root 258 objectid 38038642 offset 393216 count 1
	item 128 key (16291241984 EXTENT_ITEM 12288) itemoff 9446 itemsize 53
		refs 1 gen 456827 flags DATA
		(178 0x29740e4d67428ea8) extent data backref root 257 objectid 28407918 offset 0 count 1
	item 129 key (16291254272 EXTENT_ITEM 8192) itemoff 9393 itemsize 53
		refs 1 gen 437733 flags DATA
		(178 0x44d64df97ca8e82b) extent data backref root 258 objectid 4884981 offset 84983808 count 1
#5 lore raw
Quoted 6 lines · Russell Haley
Quoted 1 line · Qu Wenruo

Hope you still have the fs at hand.

I do, but I didn't image it, and it's been in active use for the last week. I've dumped what you suggested while running kernel 6.15.5.

In active use means all the important log trees are already gone.

Quoted 22 lines
Quoted 6 lines

In that case, I believe the following dump would help Filipe to pin down the problem:

# btrfs ins dump-tree -b 202645028864 --follow <device>
# btrfs ins dump-tree -b 202650877952 --follow <device>

These begin with

Quoted 1 line

leaf 202645028864 items 358 free space 329 generation 460240 owner CSUM_TREE

and

Quoted 1 line

leaf 202650877952 items 130 free space 6143 generation 460240 owner EXTENT_TREE

Owner not being LOG_TREE means those blocks have been reused and the evidence is gone? Attached just in case it's still useful, but I suspect not.

Quoted 2 lines
# btrfs ins dump-tree -t 256 <device>
# btrfs ins dump-tree -t 258 <device>
Quoted 7 lines

These are quite large. The first is 17 MB and the second is 1.9 GB. Not attached, due to size and likelihood of being useless.

I'm not going to start fipping the power switch to corrupt my disk intentionally, but if it happens again, I'll collect these dumps in the immediate aftermath. I assume you formed those commands from the ROOT_ITEM and bytenr in the -t TREE_LOG dump.

Yes. If possible please dump them if you hit it again.

Thanks,
Qu

Quoted 21 lines
Quoted 9 lines

Log tree replay is a very complex mechanism, it records changes into the log tree (dumped by the first two commands).
And replaying the log means we need to use the difference (the log tree) to update the target subvolumes (dumped by the last two commands).

Problems in either tree can cause replay failure, thus you need to dump both the log trees (not only the log tree root, but the real log tree for each subvolume) and the source subvolumes.

It seems like there are two issues at play here. I posted to the list because the 6.15 kernel handled replay failure ungracefully, leaving the block device in an uninteractable state.

Until I saw the other threads about a spike in log tree corruption reports, I had assumed the corruption itself was caused by hardware misbehaving during uncontrolled mains voltage brownout.

Thanks,

Russell