Skip to contentSkip to conversation

out of space (>865 GB free), filesystem remounts read-only - how to recover?

DISCUSSION Tomasz Chmielewski 3 participants 7 messages Aug 20, 2024 – Aug 21, 2024 Btrfs file system

lore
Share & tools
lore
#0 lore raw

My 44 TB filesystem was getting full (~865 GB free out of 44 TB in total), so I thought removing some old data would be a good thing to do.

Unfortunately now I'm not able to recover - the filesystem remounts read-only (no free space in metadata) a few minutes after mounting.

- kernel is 6.10.6

- the filesystem was writable and did not have any problems

- I've decided to delete some subvolumes with "btrfs sub del ..."

- a couple of minutes later after deleting the subvolumes, the filesystem went read only

- when I unmount and mount again - it resumes deleting the subvolumes in the background, free space goes up from 865G to 870G and filesystem remounts read-only

- after unmounting/mounting again, it again has 865GB free

- even if I remove some files right after mounting, as soon as the filesystem goes read-only - I unmount/mount again - removed files magically appear again

- I've added a 100 GB device to the filesystem - but it doesn't help, the filesystem still goes read only shortly after mounting

- tried running data and metadata balance - but the filesystem still goes read only shortly after mounting

How can I recover from that?

1) dmesg after mounting with "skip_balance" option:

[26883.769459] BTRFS info (device sdc1 state EA): last unmount of filesystem a80ce575-8c39-4065-80ce-2ca015fa1d51
[26890.241370] BTRFS info (device sdc1): first mount of filesystem a80ce575-8c39-4065-80ce-2ca015fa1d51
[26890.241402] BTRFS info (device sdc1): using crc32c (crc32c-intel) checksum algorithm
[26890.241414] BTRFS info (device sdc1): using free-space-tree
[27025.188366] BTRFS info (device sdc1): balance: resume skipped

2) dmesg after it goes read-only:

[27493.879003] ------------[ cut here ]------------
[27493.879014] BTRFS: Transaction aborted (error -28)
[27493.887226] BTRFS: error (device sdc1 state A) in do_free_extent_accounting:2968: errno=-28 No space left
[27493.936410] WARNING: CPU: 7 PID: 2990 at fs/btrfs/extent-tree.c:2968 do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.050834] BTRFS info (device sdc1 state EA): forced readonly
[27494.050841] BTRFS error (device sdc1 state EA): failed to run delayed ref for logical 7329684533248 num_bytes 53248 type 184 action 2 ref_mod 1: -28
[27494.051372] Modules linked in: tls
[27494.052220] BTRFS: error (device sdc1 state EA) in btrfs_run_delayed_refs:2207: errno=-28 No space left
[27494.211400]  isofs binfmt_misc ip6t_REJECT nf_reject_ipv6 nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm rapl intel_cstate cmdlinepart spi_nor input_leds mei_me joydev intel_pch_thermal mtd mei ioatdma acpi_ipmi mac_hid ipmi_si ipmi_devintf ipmi_msghandler acpi_pad sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul polyval_clmulni spi_intel_platform polyval_generic ixgbe ghash_clmulni_intel spi_intel ahci gpio_ich igb i2c_i801 xfrm_algo sha256_ssse3 mxm_wmi i2c_mux libahci lpc_ich sha1_ssse3 i2c_smbus i2c_algo_bit xhci_pci
[27494.325621]  dca xhci_pci_renesas mdio wmi aesni_intel crypto_simd cryptd
[27494.325631] CPU: 7 PID: 2990 Comm: btrfs-transacti Tainted: G        W          6.10.6-061006-generic #202408190440
[27494.325635] Hardware name: Supermicro Super Server/X10SDV-TLN4F, BIOS 2.1.v1 06/05/2020
[27494.325637] RIP: 0010:do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.325721] Code: 83 e0 01 e8 ec 52 00 00 e9 d0 5f f1 ff 89 df e8 e0 ee ff ff 84 c0 0f 84 e8 00 00 00 89 de 48 c7 c7 d0 4f a4 c0 e8 4a c4 b7 fb <0f> 0b eb bc 45 31 c0 41 83 e0 01 89 d9 ba b1 0b 00 00 4c 89 e7 48
[27494.325724] RSP: 0018:ffff956e01fbbb50 EFLAGS: 00010246
[27494.325727] RAX: 0000000000000000 RBX: 00000000ffffffe4 RCX: 0000000000000000
[27494.325729] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[27494.325731] RBP: ffff956e01fbbb80 R08: 0000000000000000 R09: 0000000000000000
[27494.325733] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8a4135049f18
[27494.325735] R13: 000006aa92150000 R14: 0000000000010000 R15: ffff956e01fbbc08
[27494.325738] FS:  0000000000000000(0000) GS:ffff8a479f980000(0000) knlGS:0000000000000000
[27494.325740] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[27494.325742] CR2: 0000713914aeef04 CR3: 000000016223c002 CR4: 00000000003706f0
[27494.325745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[27494.325747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[27494.325749] Call Trace:
[27494.325751]  <TASK>
[27494.325754]  ? show_trace_log_lvl+0x1be/0x310
[27494.325761]  ? show_trace_log_lvl+0x1be/0x310
[27494.325766]  ? __btrfs_free_extent.isra.0+0x62a/0x900 [btrfs]
[27494.325838]  ? show_regs.part.0+0x22/0x30
[27494.325842]  ? show_regs.cold+0x8/0x10
[27494.325845]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.325928]  ? __warn.cold+0xa7/0x101
[27494.325934]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.326010]  ? report_bug+0x114/0x160
[27494.326015]  ? irq_work_queue+0x2d/0x70
[27494.326021]  ? handle_bug+0x51/0xa0
[27494.326025]  ? exc_invalid_op+0x18/0x80
[27494.326028]  ? asm_exc_invalid_op+0x1b/0x20
[27494.326034]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.326109]  __btrfs_free_extent.isra.0+0x62a/0x900 [btrfs]
[27494.326181]  run_delayed_data_ref+0x6c/0x1d0 [btrfs]
[27494.326247]  btrfs_run_delayed_refs_for_head+0x30a/0x420 [btrfs]
[27494.326312]  __btrfs_run_delayed_refs+0x106/0x1b0 [btrfs]
[27494.326375]  btrfs_run_delayed_refs+0x3c/0xd0 [btrfs]
[27494.326437]  btrfs_commit_transaction+0x6a/0xc70 [btrfs]
[27494.326513]  transaction_kthread+0x167/0x1d0 [btrfs]
[27494.326584]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[27494.326653]  kthread+0xe4/0x110
[27494.326657]  ? __pfx_kthread+0x10/0x10
[27494.326660]  ret_from_fork+0x47/0x70
[27494.326665]  ? __pfx_kthread+0x10/0x10
[27494.326667]  ret_from_fork_asm+0x1a/0x30
[27494.326673]  </TASK>
[27494.326674] ---[ end trace 0000000000000000 ]---
[27494.326678] BTRFS info (device sdc1 state EA): dumping space info:
[27494.326681] BTRFS info (device sdc1 state EA): space_info DATA has 928231587840 free, is not full
[27494.326684] BTRFS info (device sdc1 state EA): space_info total=35882286448640, used=34948418105344, pinned=5635772416, reserved=0, may_use=0, readonly=983040 zone_unusable=0
[27494.326688] BTRFS info (device sdc1 state EA): space_info METADATA has -6576521216 free, is full
[27494.326691] BTRFS info (device sdc1 state EA): space_info total=78383153152, used=77616775168, pinned=71188480, reserved=695123968, may_use=6576521216, readonly=65536 zone_unusable=0
[27494.326695] BTRFS info (device sdc1 state EA): space_info SYSTEM has 5980160 free, is not full
[27494.326698] BTRFS info (device sdc1 state EA): space_info total=8388608, used=2408448, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
[27494.326701] BTRFS info (device sdc1 state EA): global_block_rsv: size 536870912 reserved 536805376
[27494.326703] BTRFS info (device sdc1 state EA): trans_block_rsv: size 0 reserved 0
[27494.326706] BTRFS info (device sdc1 state EA): chunk_block_rsv: size 0 reserved 0
[27494.326708] BTRFS info (device sdc1 state EA): delayed_block_rsv: size 0 reserved 0
[27494.326710] BTRFS info (device sdc1 state EA): delayed_refs_rsv: size 1727549669376 reserved 6039715840
[27494.326713] BTRFS: error (device sdc1 state EA) in do_free_extent_accounting:2968: errno=-28 No space left
[27494.443071] BTRFS error (device sdc1 state EA): failed to run delayed ref for logical 7329665056768 num_bytes 65536 type 184 action 2 ref_mod 1: -28
[27494.603135] BTRFS: error (device sdc1 state EA) in btrfs_run_delayed_refs:2207: errno=-28 No space left

Tomasz Chmielewski

#1 lore raw
Quoted 2 lines · Tomasz Chmielewski

My 44 TB filesystem was getting full (~865 GB free out of 44 TB in total), so I thought removing some old data would be a good thing to do.

btrfs fi usage output please.

I bet it's some btrfs RAID setup, maybe RAID1/10 or RAID1C3/C4 for metadata. And the disk usages got unbalanced, the metadata overcommitment did a wrong estimation and cause the ENOSPC.

Thanks,
Qu

Quoted 174 lines

Unfortunately now I'm not able to recover - the filesystem remounts read-only (no free space in metadata) a few minutes after mounting.

- kernel is 6.10.6

- the filesystem was writable and did not have any problems

- I've decided to delete some subvolumes with "btrfs sub del ..."

- a couple of minutes later after deleting the subvolumes, the filesystem went read only

- when I unmount and mount again - it resumes deleting the subvolumes in the background, free space goes up from 865G to 870G and filesystem remounts read-only

- after unmounting/mounting again, it again has 865GB free

- even if I remove some files right after mounting, as soon as the filesystem goes read-only - I unmount/mount again - removed files magically appear again

- I've added a 100 GB device to the filesystem - but it doesn't help, the filesystem still goes read only shortly after mounting

- tried running data and metadata balance - but the filesystem still goes read only shortly after mounting

How can I recover from that?

1) dmesg after mounting with "skip_balance" option:

[26883.769459] BTRFS info (device sdc1 state EA): last unmount of
filesystem a80ce575-8c39-4065-80ce-2ca015fa1d51
[26890.241370] BTRFS info (device sdc1): first mount of filesystem
a80ce575-8c39-4065-80ce-2ca015fa1d51
[26890.241402] BTRFS info (device sdc1): using crc32c (crc32c-intel)
checksum algorithm
[26890.241414] BTRFS info (device sdc1): using free-space-tree
[27025.188366] BTRFS info (device sdc1): balance: resume skipped

2) dmesg after it goes read-only:

[27493.879003] ------------[ cut here ]------------
[27493.879014] BTRFS: Transaction aborted (error -28)
[27493.887226] BTRFS: error (device sdc1 state A) in
do_free_extent_accounting:2968: errno=-28 No space left
[27493.936410] WARNING: CPU: 7 PID: 2990 at fs/btrfs/extent-tree.c:2968
do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.050834] BTRFS info (device sdc1 state EA): forced readonly
[27494.050841] BTRFS error (device sdc1 state EA): failed to run delayed
ref for logical 7329684533248 num_bytes 53248 type 184 action 2 ref_mod
1: -28
[27494.051372] Modules linked in: tls
[27494.052220] BTRFS: error (device sdc1 state EA) in
btrfs_run_delayed_refs:2207: errno=-28 No space left
[27494.211400]  isofs binfmt_misc ip6t_REJECT nf_reject_ipv6

nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm rapl intel_cstate cmdlinepart spi_nor input_leds mei_me joydev intel_pch_thermal mtd mei ioatdma acpi_ipmi mac_hid ipmi_si ipmi_devintf ipmi_msghandler acpi_pad sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul polyval_clmulni spi_intel_platform polyval_generic ixgbe ghash_clmulni_intel spi_intel ahci gpio_ich igb i2c_i801 xfrm_algo sha256_ssse3 mxm_wmi i2c_mux libahci lpc_ich sha1_ssse3 i2c_smbus i2c_algo_bit xhci_pci

[27494.325621]  dca xhci_pci_renesas mdio wmi aesni_intel crypto_simd
cryptd
[27494.325631] CPU: 7 PID: 2990 Comm: btrfs-transacti Tainted: G
W          6.10.6-061006-generic #202408190440
[27494.325635] Hardware name: Supermicro Super Server/X10SDV-TLN4F, BIOS
2.1.v1 06/05/2020
[27494.325637] RIP: 0010:do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.325721] Code: 83 e0 01 e8 ec 52 00 00 e9 d0 5f f1 ff 89 df e8 e0
ee ff ff 84 c0 0f 84 e8 00 00 00 89 de 48 c7 c7 d0 4f a4 c0 e8 4a c4 b7
fb <0f> 0b eb bc 45 31 c0 41 83 e0 01 89 d9 ba b1 0b 00 00 4c 89 e7 48
[27494.325724] RSP: 0018:ffff956e01fbbb50 EFLAGS: 00010246
[27494.325727] RAX: 0000000000000000 RBX: 00000000ffffffe4 RCX:
0000000000000000
[27494.325729] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[27494.325731] RBP: ffff956e01fbbb80 R08: 0000000000000000 R09:
0000000000000000
[27494.325733] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff8a4135049f18
[27494.325735] R13: 000006aa92150000 R14: 0000000000010000 R15:
ffff956e01fbbc08
[27494.325738] FS:  0000000000000000(0000) GS:ffff8a479f980000(0000)
knlGS:0000000000000000
[27494.325740] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[27494.325742] CR2: 0000713914aeef04 CR3: 000000016223c002 CR4:
00000000003706f0
[27494.325745] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[27494.325747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[27494.325749] Call Trace:
[27494.325751]  <TASK>
[27494.325754]  ? show_trace_log_lvl+0x1be/0x310
[27494.325761]  ? show_trace_log_lvl+0x1be/0x310
[27494.325766]  ? __btrfs_free_extent.isra.0+0x62a/0x900 [btrfs]
[27494.325838]  ? show_regs.part.0+0x22/0x30
[27494.325842]  ? show_regs.cold+0x8/0x10
[27494.325845]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.325928]  ? __warn.cold+0xa7/0x101
[27494.325934]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.326010]  ? report_bug+0x114/0x160
[27494.326015]  ? irq_work_queue+0x2d/0x70
[27494.326021]  ? handle_bug+0x51/0xa0
[27494.326025]  ? exc_invalid_op+0x18/0x80
[27494.326028]  ? asm_exc_invalid_op+0x1b/0x20
[27494.326034]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.326109]  __btrfs_free_extent.isra.0+0x62a/0x900 [btrfs]
[27494.326181]  run_delayed_data_ref+0x6c/0x1d0 [btrfs]
[27494.326247]  btrfs_run_delayed_refs_for_head+0x30a/0x420 [btrfs]
[27494.326312]  __btrfs_run_delayed_refs+0x106/0x1b0 [btrfs]
[27494.326375]  btrfs_run_delayed_refs+0x3c/0xd0 [btrfs]
[27494.326437]  btrfs_commit_transaction+0x6a/0xc70 [btrfs]
[27494.326513]  transaction_kthread+0x167/0x1d0 [btrfs]
[27494.326584]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[27494.326653]  kthread+0xe4/0x110
[27494.326657]  ? __pfx_kthread+0x10/0x10
[27494.326660]  ret_from_fork+0x47/0x70
[27494.326665]  ? __pfx_kthread+0x10/0x10
[27494.326667]  ret_from_fork_asm+0x1a/0x30
[27494.326673]  </TASK>
[27494.326674] ---[ end trace 0000000000000000 ]---
[27494.326678] BTRFS info (device sdc1 state EA): dumping space info:
[27494.326681] BTRFS info (device sdc1 state EA): space_info DATA has
928231587840 free, is not full
[27494.326684] BTRFS info (device sdc1 state EA): space_info
total=35882286448640, used=34948418105344, pinned=5635772416,
reserved=0, may_use=0, readonly=983040 zone_unusable=0
[27494.326688] BTRFS info (device sdc1 state EA): space_info METADATA
has -6576521216 free, is full
[27494.326691] BTRFS info (device sdc1 state EA): space_info
total=78383153152, used=77616775168, pinned=71188480,
reserved=695123968, may_use=6576521216, readonly=65536 zone_unusable=0
[27494.326695] BTRFS info (device sdc1 state EA): space_info SYSTEM has
5980160 free, is not full
[27494.326698] BTRFS info (device sdc1 state EA): space_info
total=8388608, used=2408448, pinned=0, reserved=0, may_use=0, readonly=0
zone_unusable=0
[27494.326701] BTRFS info (device sdc1 state EA): global_block_rsv: size
536870912 reserved 536805376
[27494.326703] BTRFS info (device sdc1 state EA): trans_block_rsv: size
0 reserved 0
[27494.326706] BTRFS info (device sdc1 state EA): chunk_block_rsv: size
0 reserved 0
[27494.326708] BTRFS info (device sdc1 state EA): delayed_block_rsv:
size 0 reserved 0
[27494.326710] BTRFS info (device sdc1 state EA): delayed_refs_rsv: size
1727549669376 reserved 6039715840
[27494.326713] BTRFS: error (device sdc1 state EA) in
do_free_extent_accounting:2968: errno=-28 No space left
[27494.443071] BTRFS error (device sdc1 state EA): failed to run delayed
ref for logical 7329665056768 num_bytes 65536 type 184 action 2 ref_mod
1: -28
[27494.603135] BTRFS: error (device sdc1 state EA) in

btrfs_run_delayed_refs:2207: errno=-28 No space left

Tomasz Chmielewski

#2 lore raw
Quoted 9 lines · Qu Wenruo
Quoted 2 lines · Tomasz Chmielewski

My 44 TB filesystem was getting full (~865 GB free out of 44 TB in total), so I thought removing some old data would be a good thing to do.

btrfs fi usage output please.

I bet it's some btrfs RAID setup, maybe RAID1/10 or RAID1C3/C4 for metadata. And the disk usages got unbalanced, the metadata overcommitment did a wrong estimation and cause the ENOSPC.

# btrfs filesystem usage /data1
Overall:
    Device size:                  43.75TiB
    Device allocated:             43.66TiB
    Device unallocated:          100.00GiB
    Device missing:                  0.00B
    Used:                         42.51TiB
    Free (estimated):            951.71GiB      (min: 926.71GiB)
    Free (statfs, df):           876.71GiB
    Data ratio:                       1.33
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 64.00KiB)
    Multiple profiles:                  no

Data,RAID5: Size:32.63TiB, Used:31.78TiB (97.38%)

   /dev/sdc1      10.88TiB
   /dev/sdd1      10.88TiB
   /dev/sde1      10.88TiB
   /dev/sdf1      10.88TiB

Metadata,RAID1: Size:73.00GiB, Used:71.89GiB (98.48%)

   /dev/sdc1      37.00GiB
   /dev/sdd1      37.00GiB
   /dev/sde1      36.00GiB
   /dev/sdf1      36.00GiB

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
/dev/sde1 8.00MiB
/dev/sdf1 8.00MiB

Unallocated:

   /dev/sdc1       1.00MiB
   /dev/sdd1       1.00MiB
   /dev/sde1       1.00MiB
   /dev/sdf1       1.00MiB
   /dev/loop0    100.00GiB
# btrfs filesystem usage /data1
Overall:
    Device size:                  43.75TiB
    Device allocated:             43.66TiB
    Device unallocated:          100.00GiB
    Device missing:                  0.00B
    Used:                         42.51TiB
    Free (estimated):            951.71GiB      (min: 926.71GiB)
    Free (statfs, df):           876.71GiB
    Data ratio:                       1.33
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 64.00KiB)
    Multiple profiles:                  no

Data,RAID5: Size:32.63TiB, Used:31.78TiB (97.38%)

   /dev/sdc1      10.88TiB
   /dev/sdd1      10.88TiB
   /dev/sde1      10.88TiB
   /dev/sdf1      10.88TiB

Metadata,RAID1: Size:73.00GiB, Used:71.89GiB (98.48%)

   /dev/sdc1      37.00GiB
   /dev/sdd1      37.00GiB
   /dev/sde1      36.00GiB
   /dev/sdf1      36.00GiB

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
/dev/sde1 8.00MiB
/dev/sdf1 8.00MiB

Unallocated:

   /dev/sdc1       1.00MiB
   /dev/sdd1       1.00MiB
   /dev/sde1       1.00MiB
   /dev/sdf1       1.00MiB

/dev/loop0 100.00GiB

# btrfs filesystem df /data1
Data, RAID5: total=32.63TiB, used=31.78TiB
System, RAID1: total=8.00MiB, used=2.30MiB
Metadata, RAID1: total=73.00GiB, used=71.89GiB
GlobalReserve, single: total=512.00MiB, used=64.00KiB
# btrfs filesystem show /data1
Label: 'data'  uuid: a80ce575-8c39-4065-80ce-2ca015fa1d51
        Total devices 5 FS bytes used 31.85TiB
        devid    1 size 10.91TiB used 10.91TiB path /dev/sdc1
        devid    2 size 10.91TiB used 10.91TiB path /dev/sdd1
        devid    3 size 10.91TiB used 10.91TiB path /dev/sde1
        devid    4 size 10.91TiB used 10.91TiB path /dev/sdf1
        devid    5 size 100.00GiB used 0.00B path /dev/loop0

Tomasz

#3 lore raw
Quoted 67 lines · Tomasz Chmielewski
Quoted 9 lines · Qu Wenruo
Quoted 2 lines · Tomasz Chmielewski

My 44 TB filesystem was getting full (~865 GB free out of 44 TB in total), so I thought removing some old data would be a good thing to do.

btrfs fi usage output please.

I bet it's some btrfs RAID setup, maybe RAID1/10 or RAID1C3/C4 for metadata. And the disk usages got unbalanced, the metadata overcommitment did a wrong estimation and cause the ENOSPC.

# btrfs filesystem usage /data1
Overall:
     Device size:                  43.75TiB
     Device allocated:             43.66TiB
     Device unallocated:          100.00GiB
     Device missing:                  0.00B
     Used:                         42.51TiB
     Free (estimated):            951.71GiB      (min: 926.71GiB)
     Free (statfs, df):           876.71GiB
     Data ratio:                       1.33
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 64.00KiB)
     Multiple profiles:                  no

Data,RAID5: Size:32.63TiB, Used:31.78TiB (97.38%)
   /dev/sdc1      10.88TiB
   /dev/sdd1      10.88TiB
   /dev/sde1      10.88TiB
   /dev/sdf1      10.88TiB

Metadata,RAID1: Size:73.00GiB, Used:71.89GiB (98.48%)
   /dev/sdc1      37.00GiB
   /dev/sdd1      37.00GiB
   /dev/sde1      36.00GiB
   /dev/sdf1      36.00GiB

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
   /dev/sde1       8.00MiB
   /dev/sdf1       8.00MiB

Unallocated:
   /dev/sdc1       1.00MiB
   /dev/sdd1       1.00MiB
   /dev/sde1       1.00MiB
   /dev/sdf1       1.00MiB
   /dev/loop0    100.00GiB

# btrfs filesystem usage /data1
Overall:
     Device size:                  43.75TiB
     Device allocated:             43.66TiB
     Device unallocated:          100.00GiB
     Device missing:                  0.00B
     Used:                         42.51TiB
     Free (estimated):            951.71GiB      (min: 926.71GiB)
     Free (statfs, df):           876.71GiB
     Data ratio:                       1.33
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 64.00KiB)
     Multiple profiles:                  no

Data,RAID5: Size:32.63TiB, Used:31.78TiB (97.38%)
   /dev/sdc1      10.88TiB
   /dev/sdd1      10.88TiB
   /dev/sde1      10.88TiB
   /dev/sdf1      10.88TiB

Quoted 5 lines

Metadata,RAID1: Size:73.00GiB, Used:71.89GiB (98.48%)
   /dev/sdc1      37.00GiB
   /dev/sdd1      37.00GiB
   /dev/sde1      36.00GiB
   /dev/sdf1      36.00GiB

Your metadata is obviously near full.

But it's still a bug that btrfs free space reservation code didn't return ENOSPC early, other than hitting ENOSPC at transaction commit time.

Quoted 10 lines

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
   /dev/sde1       8.00MiB
   /dev/sdf1       8.00MiB

Unallocated:
   /dev/sdc1       1.00MiB
   /dev/sdd1       1.00MiB
   /dev/sde1       1.00MiB
   /dev/sdf1       1.00MiB

At least one good news is, all your four devices are usage are very balanced.

Quoted 1 line

   /dev/loop0    100.00GiB

You just need to add another device to fulfill the RAID1 requirement.

Then you will still need to remove enough data to get at least 1GiB for two devices, then you can go balance data to free some space for your metadata.

Thanks,
Qu

Quoted 21 lines

# btrfs filesystem df /data1
Data, RAID5: total=32.63TiB, used=31.78TiB
System, RAID1: total=8.00MiB, used=2.30MiB
Metadata, RAID1: total=73.00GiB, used=71.89GiB
GlobalReserve, single: total=512.00MiB, used=64.00KiB
# btrfs filesystem show /data1
Label: 'data'  uuid: a80ce575-8c39-4065-80ce-2ca015fa1d51
         Total devices 5 FS bytes used 31.85TiB
         devid    1 size 10.91TiB used 10.91TiB path /dev/sdc1
         devid    2 size 10.91TiB used 10.91TiB path /dev/sdd1
         devid    3 size 10.91TiB used 10.91TiB path /dev/sde1
         devid    4 size 10.91TiB used 10.91TiB path /dev/sdf1
         devid    5 size 100.00GiB used 0.00B path /dev/loop0

Tomasz

#4 lore raw
Quoted 10 lines · Qu Wenruo
Quoted 9 lines

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
   /dev/sde1       8.00MiB
   /dev/sdf1       8.00MiB

Unallocated:
   /dev/sdc1       1.00MiB
   /dev/sdd1       1.00MiB
   /dev/sde1       1.00MiB
   /dev/sdf1       1.00MiB

Quoted 4 lines

At least one good news is, all your four devices are usage are very balanced.

Quoted 1 line

   /dev/loop0    100.00GiB

You just need to add another device to fulfill the RAID1 requirement.

So, for example, one more loop device of 100 GB?

Quoted 1 line

Then you will still need to remove enough data to get at least 1GiB for two devices, then you can go balance data to free some space for your metadata.

Tomasz

#5 lore raw
Quoted 12 lines · Tomasz Chmielewski
Quoted 10 lines · Qu Wenruo
Quoted 9 lines

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
    /dev/sde1       8.00MiB
    /dev/sdf1       8.00MiB

Unallocated:
    /dev/sdc1       1.00MiB
    /dev/sdd1       1.00MiB
    /dev/sde1       1.00MiB
    /dev/sdf1       1.00MiB

Quoted 6 lines
Quoted 4 lines

At least one good news is, all your four devices are usage are very balanced.

Quoted 1 line

    /dev/loop0    100.00GiB

You just need to add another device to fulfill the RAID1 requirement.

So, for example, one more loop device of 100 GB?

Yep.

Thanks,
Qu

Quoted 6 lines
Quoted 3 lines

Then you will still need to remove enough data to get at least 1GiB for two devices, then you can go balance data to free some space for your metadata.

Tomasz

#6 lore raw
Quoted 13 lines · Qu Wenruo
Quoted 12 lines · Tomasz Chmielewski
Quoted 10 lines · Qu Wenruo
Quoted 9 lines

System,RAID1: Size:8.00MiB, Used:2.30MiB (28.71%)
    /dev/sde1       8.00MiB
    /dev/sdf1       8.00MiB

Unallocated:
    /dev/sdc1       1.00MiB
    /dev/sdd1       1.00MiB
    /dev/sde1       1.00MiB
    /dev/sdf1       1.00MiB

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

At least one good news is, all your four devices are usage are very balanced.

Quoted 1 line

    /dev/loop0    100.00GiB

You just need to add another device to fulfill the RAID1 requirement.

So, for example, one more loop device of 100 GB?

Yep.

It worked, thank you!

Tomasz