Btrfs ENOSPC / Stuck in RO with "exclusive operation balance paused in progress"
Hello,
On Debian Forky (testing), I tried to migrate my old SSD btrfs partition metadata from the previous default Single to the new DUP default.
The migration stopped halfway far /var and I am now stuck.
I am facing a severe ENOSPC deadlock it seems.
Symptoms:
- The filesystem conversion stopped midway and turned Read-Only during a balance operation.
- Attempting to remount or mount as 'rw' fails with ENOSPC.
- Adding a temporary loop device in RAM via 'btrfs device add' fails because the FS is read-only.
- Trying to run 'btrfs check --clear-space-cache v2' or cleanup tools at cold state fails with: "error: please finish/cancel the running replace/balance before running this command".
- 'btrfs balance cancel' cannot be run because mounting 'rw' is rejected, and it refuses to target the unmounted block device directly.
I can only boot into init=/bin/sh as of now.
Here are the outputs:
btrfs fi df /var
Data, single: total=65.44GiB, used=49.21GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=3.00GiB, used=1.59GiB
GlobalReserve, single: total=181.70MiB, used=0.00B
prahal ~ sudo btrfs balance start -mprofiles=single -mconvert=dup /var
ERROR: error during balancing '/var': Read-only file system
There may be more info in syslog - try dmesg | tail
uin 09 02:34:34 hermes kernel: BTRFS info (device sda3): balance: start -mconvert=dup,profiles=single -sconvert=dup,profiles=single
juin 09 02:34:34 hermes kernel: BTRFS info (device sda3): relocating block group 102442729472 flags metadata
juin 09 02:35:02 hermes kernel: BTRFS error (device sda3): failed to run delayed ref for logical 3976462336 num_bytes 8192 type 184 action 1 ref_mod 1: -28
juin 09 02:35:02 hermes kernel: ------------[ cut here ]------------
juin 09 02:35:02 hermes kernel: BTRFS: Transaction aborted (error -28)
juin 09 02:35:02 hermes kernel: WARNING: fs/btrfs/extent-tree.c:2247 at btrfs_run_delayed_refs+0x115/0x120 [btrfs], CPU#0: kworker/u32:11/1370737
juin 09 02:35:02 hermes kernel: Modules linked in: nilfs2 ufs qnx4 hfsplus hfs minix msdos jfs nls_ucs2_utils xfs ext4 mbcache jbd2 crc32c_cryptoapi cpuid exfat uinput rfcomm snd_seq_dummy snd_hrtimer snd_seq x> juin 09 02:35:02 hermes kernel: snd_soc_avs nls_ascii snd_soc_hda_codec kvm_intel snd_hda_ext_core nls_cp437 snd_hda_codec vfat snd_hda_core fat kvm btrtl btbcm snd_intel_dspcfg snd_intel_sdw_acpi snd_hwdep gs> juin 09 02:35:02 hermes kernel: hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper sd_mod cec rc_core iTCO_wdt drm_client_lib intel_pmc_bxt xhci_pci watchdog> juin 09 02:35:02 hermes kernel: CPU: 0 UID: 0 PID: 1370737 Comm: kworker/u32:11 Tainted: P OE 7.0.9+deb14-amd64 #1 PREEMPT(lazy) Debian 7.0.9-1
juin 09 02:35:02 hermes kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
juin 09 02:35:02 hermes kernel: Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
juin 09 02:35:02 hermes kernel: Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
juin 09 02:35:02 hermes kernel: RIP: 0010:btrfs_run_delayed_refs+0x117/0x120 [btrfs]
juin 09 02:35:02 hermes kernel: Code: 32 27 c1 48 89 ef e8 e8 54 0f 00 eb 07 e8 f1 22 ed d5 31 db 89 d8 5b 5d 41 5c 41 5d c3 cc cc cc cc 48 8d 3d 5b 7f 31 00 89 de <67> 48 0f b9 3a eb be 66 90 90 90 90 90 90 90> juin 09 02:35:02 hermes kernel: RSP: 0018:ffffd03f75b87d48 EFLAGS: 00010292
juin 09 02:35:02 hermes kernel: RAX: 0000000000000002 RBX: 00000000ffffffe4 RCX: ffff8ad782c55600
juin 09 02:35:02 hermes kernel: RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc11dcac0
juin 09 02:35:02 hermes kernel: RBP: ffff8ad783d7bc78 R08: 0000000000000000 R09: ffffffffc0ec412a
juin 09 02:35:02 hermes kernel: R10: fffff4c487b11b40 R11: ffff8ad86c46d460 R12: 0000000000000000
juin 09 02:35:02 hermes kernel: R13: ffff8ad6a2573c00 R14: 0000000000001252 R15: 0000000000000000
juin 09 02:35:02 hermes kernel: FS: 0000000000000000(0000) GS:ffff8adee617c000(0000) knlGS:0000000000000000
juin 09 02:35:02 hermes kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
juin 09 02:35:02 hermes kernel: CR2: 00007ffd95039e68 CR3: 000000072ca2c005 CR4: 00000000003726f0
juin 09 02:35:02 hermes kernel: Call Trace:
juin 09 02:35:02 hermes kernel: <TASK>
juin 09 02:35:02 hermes kernel: flush_space+0x126/0x5a0 [btrfs]
juin 09 02:35:02 hermes kernel: ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
juin 09 02:35:02 hermes kernel: ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
juin 09 02:35:02 hermes kernel: do_async_reclaim_metadata_space+0x7d/0x180 [btrfs]
juin 09 02:35:02 hermes kernel: btrfs_async_reclaim_metadata_space+0x54/0x70 [btrfs]
juin 09 02:35:02 hermes kernel: process_one_work+0x19d/0x3a0
juin 09 02:35:02 hermes kernel: worker_thread+0x1af/0x320
juin 09 02:35:02 hermes kernel: ? __pfx_worker_thread+0x10/0x10
juin 09 02:35:02 hermes kernel: kthread+0xe3/0x120
juin 09 02:35:02 hermes kernel: ? __pfx_kthread+0x10/0x10
juin 09 02:35:02 hermes kernel: ret_from_fork+0x2c7/0x350
juin 09 02:35:02 hermes kernel: ? __pfx_kthread+0x10/0x10
juin 09 02:35:02 hermes kernel: ret_from_fork_asm+0x1a/0x30
juin 09 02:35:02 hermes kernel: </TASK>
juin 09 02:35:02 hermes kernel: ---[ end trace 0000000000000000 ]---
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): dumping space info:
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info DATA (sub-group id 0) has 17422733312 free, is full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=192512, readonly=65536 zone_unusable=0 juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info METADATA (sub-group id 0) has 442368 free, is full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=3221225472, used=2155544576, pinned=38010880, reserved=10387456, may_use=576634880, readonly=440205312 zone_unusable=0 juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info SYSTEM (sub-group id 0) has 33538048 free, is not full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): global_block_rsv: size 190529536 reserved 190480384
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): trans_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): chunk_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): remap_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): delayed_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): delayed_refs_rsv: size 1270611968 reserved 368197632
juin 09 02:35:02 hermes kernel: BTRFS: error (device sda3 state A) in btrfs_run_delayed_refs:2247: errno=-28 No space left
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): forced readonly
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): 1 enospc errors during balance
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): balance: ended with status: -30
Linux (none) 7.0.10+deb14-amd64 #1 SMP PREEMPT_DYNAMIC Debian 7.0.10-1 (2026-05-27) x86_64 GNU/Linux
btrfs-progs v7.0-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> LABEL=ROOT / btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2 0 1 UUID=9C79-15CA /boot/efi vfat umask=0077 0 1 LABEL=HDDHOME /home btrfs defaults,compress=zstd,noatime,space_cache=v2 0 2 #LABEL=SSDHOME /ssdhome btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2 0 2 LABEL=SSDHOME /var btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2,subvol=@var 0 2 UUID=EAB2-8BE4 /boot/efi-old vfat umask=0077 0 1
btrfs fi usage /var/@var/
Overall:
Device size: 68.47GiB Device allocated: 68.47GiB Device unallocated: 1.00MiB Device missing: 0.00B Device slack: 0.00B Used: 51.21GiB Free (estimated): 16.23GiB (min: 16.23GiB) Free (statfs, df): 16.23GiB Data ratio: 1.00 Metadata ratio: 1.00 Global reserve: 95.19MiB (used: 0.00B) Multiple profiles: no
Data,single: Size:65.44GiB, Used:49.21GiB (75.20%)
/dev/sda3 65.44GiB
Metadata,single: Size:3.00GiB, Used:2.00GiB (66.51%)
/dev/sda3 3.00GiB
System,single: Size:32.00MiB, Used:16.00KiB (0.05%)
/dev/sda3 32.00MiB
Unallocated:
/dev/sda3 1.00MiB
btrfs balance status /var/@var/
Balance on '/var/@var/' is paused
0 out of about 0 chunks balanced (0 considered), -nan% left
btrfs balance cancel /var/@var/
ERROR: balance cancel on '/var/@var/' failed: Read-only file system
Thank you for your help,
Alban
more details after a reboot (sda3 is now sdb3):
# mount -o ro,rescue=all /dev/sdb3 /var [ 597.965956] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (437) [ 597.977243] BTRFS info (device sdb3 state S): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [ 597.987580] BTRFS info (device sdb3 state S): using crc32c checksum algorithm [ 598.028410] BTRFS info (device sdb3 state ECS): enabling ssd optimizations [ 598.035833] BTRFS info (device sdb3 state ECS): disabling log replay at mount time [ 598.043944] BTRFS info (device sdb3 state ECS): turning on async discard [ 598.051195] BTRFS info (device sdb3 state ECS): enabling free space tree [ 598.058445] BTRFS info (device sdb3 state ECS): ignoring bad roots [ 598.065110] BTRFS info (device sdb3 state ECS): ignoring data csums [ 598.071836] BTRFS info (device sdb3 state ECS): ignoring meta csums [ 598.078558] BTRFS info (device sdb3 state ECS): ignoring unknown super block flags
# btrfs balance resume /var ERROR: error during balancing '/var': Read-only file system There may be more info in syslog - try dmesg | tail
# umount /var [ 1077.443570] BTRFS info (device sdb3 state EMCS): last unmount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
# mount -o rw,skip_balance,clear_cache /dev/sdb3 /var
[ 1084.400924] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (464)
[ 1084.412230] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[ 1084.421817] BTRFS info (device sdb3): using crc32c checksum algorithm
[ 1084.458776] BTRFS info (device sdb3): rebuilding free space tree
[ 1084.465327] BTRFS warning (device sdb3): Skipping commit of aborted transaction.
[ 1084.473209] ------------[ cut here ]------------
[ 1084.478283] BTRFS: Transaction aborted (error -28)
[ 1084.483526] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#6: kworker/u32:2/66
[ 1084.494986] Modules linked in: ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod autof
s4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper cec sd_mod rc_core drm_clie
nt_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 video realtek crc_itu_t usbcore scsi_m
od fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs [ 1084.546146] CPU: 6 UID: 0 PID: 66 Comm: kworker/u32:2 Not tainted 7.0.10+deb14-amd64 #1 PREEMPT(lazy) Debian 7.0.10-1
[ 1084.557401] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
[ 1084.564828] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[ 1084.572751] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs]
[ 1084.579439] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9 f
8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00
[ 1084.598660] RSP: 0018:ffffce9fc02fbce8 EFLAGS: 00010292
[ 1084.604372] RAX: 0000000000000002 RBX: ffff8ac4c75ae150 RCX: 0000000000000003
[ 1084.611984] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080
[ 1084.619598] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fc02fbad0
[ 1084.627200] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac4d02dd000
[ 1084.634799] R13: ffff8ac4d13a7600 R14: ffff8ac4d02dd4c0 R15: ffff8ac4c1100000
[ 1084.642401] FS: 0000000000000000(0000) GS:ffff8acc3a4fc000(0000) knlGS:0000000000000000
[ 1084.650947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1084.657153] CR2: 00007f59bc90839c CR3: 0000000168c2c003 CR4: 00000000003706f0
[ 1084.664742] Call Trace:
[ 1084.667646] <TASK>
[ 1084.670180] ? start_transaction+0x228/0x880 [btrfs]
[ 1084.675649] flush_space+0x3c3/0x5a0 [btrfs]
[ 1084.680430] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[ 1084.686591] ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
[ 1084.693190] do_async_reclaim_metadata_space+0x7d/0x180 [btrfs]
[ 1084.699618] btrfs_async_reclaim_metadata_space+0x54/0x70 [btrfs]
[ 1084.706214] process_one_work+0x19d/0x3a0
[ 1084.710679] worker_thread+0x1af/0x320
[ 1084.714880] ? __pfx_worker_thread+0x10/0x10
[ 1084.719592] kthread+0xe3/0x120
[ 1084.723174] ? __pfx_kthread+0x10/0x10
[ 1084.727370] ret_from_fork+0x2c7/0x350
[ 1084.731562] ? __pfx_kthread+0x10/0x10
[ 1084.735738] ret_from_fork_asm+0x1a/0x30
[ 1084.740102] </TASK>
[ 1084.742707] ---[ end trace 0000000000000000 ]---
[ 1084.747733] BTRFS info (device sdb3 state A): dumping space info:
[ 1084.754275] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full
[ 1084.764845] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[ 1084.779326] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full
[ 1084.789819] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[ 1084.805227] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[ 1084.815115] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[ 1084.829087] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080
[ 1084.838378] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0
[ 1084.846167] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0
[ 1084.853986] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0
[ 1084.861767] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0
[ 1084.869752] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0
[ 1084.877621] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left
[ 1084.887109] BTRFS warning (device sdb3 state EA): failed to rebuild free space tree: -28
[ 1084.895714] BTRFS error (device sdb3 state EA): commit super block returned -30
[ 1084.904462] BTRFS error (device sdb3 state EA): open_ctree failed: -28
mount: /var: fsconfig() failed: No space left on device.
dmesg(1) may have more information after failed mount system call.Le 09/06/2026 à 06:42, Alban Browaeys a écrit :
Quoted 167 lines
Hello,
On Debian Forky (testing), I tried to migrate my old SSD btrfs partition metadata from the previous default Single to the new DUP default.
The migration stopped halfway far /var and I am now stuck.
I am facing a severe ENOSPC deadlock it seems.
Symptoms:
- The filesystem conversion stopped midway and turned Read-Only during a balance operation.
- Attempting to remount or mount as 'rw' fails with ENOSPC.
- Adding a temporary loop device in RAM via 'btrfs device add' fails because the FS is read-only.
- Trying to run 'btrfs check --clear-space-cache v2' or cleanup tools
at cold state fails with: "error: please finish/cancel the running replace/balance before running this command".- 'btrfs balance cancel' cannot be run because mounting 'rw' is rejected, and it refuses to target the unmounted block device directly.
I can only boot into init=/bin/sh as of now.
Here are the outputs:
btrfs fi df /var
Data, single: total=65.44GiB, used=49.21GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=3.00GiB, used=1.59GiB
GlobalReserve, single: total=181.70MiB, used=0.00B
prahal ~ sudo btrfs balance start -mprofiles=single -mconvert=dup /var
ERROR: error during balancing '/var': Read-only file system
There may be more info in syslog - try dmesg | tail
uin 09 02:34:34 hermes kernel: BTRFS info (device sda3): balance: start -mconvert=dup,profiles=single -sconvert=dup,profiles=single
juin 09 02:34:34 hermes kernel: BTRFS info (device sda3): relocating block group 102442729472 flags metadata
juin 09 02:35:02 hermes kernel: BTRFS error (device sda3): failed to run delayed ref for logical 3976462336num_bytes8192 type 184 action 1ref_mod1: -28
juin 09 02:35:02 hermes kernel: ------------[ cut here ]------------
juin 09 02:35:02 hermes kernel: BTRFS: Transaction aborted (error -28)
juin 09 02:35:02 hermes kernel: WARNING:fs/btrfs/extent-tree.c:2247atbtrfs_run_delayed_refs+0x115/0x120[btrfs], CPU#0: kworker/u32:11/1370737
juin 09 02:35:02 hermes kernel: Modules linked in: nilfs2 ufs qnx4 hfsplus hfs minix msdos jfsnls_ucs2_utilsxfs ext4 mbcache jbd2crc32c_cryptoapicpuid exfat uinput rfcommsnd_seq_dummysnd_hrtimersnd_seqx> juin 09 02:35:02 hermes kernel:snd_soc_avsnls_asciisnd_soc_hda_codeckvm_intelsnd_hda_ext_corenls_cp437snd_hda_codecvfatsnd_hda_corefat kvm btrtl btbcmsnd_intel_dspcfgsnd_intel_sdw_acpisnd_hwdepgs> juin 09 02:35:02 hermes kernel:hid_genericusbhid hidsr_modcdrom uasusb_storagei915drm_buddyttmi2c_algo_bitdrm_display_helpersd_modcecrc_coreiTCO_wdtdrm_client_libintel_pmc_bxtxhci_pciwatchdog> juin 09 02:35:02 hermes kernel: CPU: 0 UID: 0 PID: 1370737 Comm: kworker/u32:11 Tainted: P OE 7.0.9+deb14-amd64 #1 PREEMPT(lazy) Debian 7.0.9-1
juin 09 02:35:02 hermes kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
juin 09 02:35:02 hermes kernel: Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
juin 09 02:35:02 hermes kernel: Workqueue:events_unboundbtrfs_async_reclaim_metadata_space[btrfs]
juin 09 02:35:02 hermes kernel: RIP: 0010:btrfs_run_delayed_refs+0x117/0x120 [btrfs]
juin 09 02:35:02 hermes kernel: Code: 32 27 c1 48 89 ef e8 e8 54 0f 00 eb 07 e8 f1 22 ed d5 31 db 89 d8 5b 5d 41 5c 41 5d c3 cc cc cc cc 48 8d 3d 5b 7f 31 00 89 de <67> 48 0f b9 3a eb be 66 90 90 90 90 90 90 90> juin 09 02:35:02 hermes kernel: RSP: 0018:ffffd03f75b87d48 EFLAGS: 00010292
juin 09 02:35:02 hermes kernel: RAX: 0000000000000002 RBX:00000000ffffffe4RCX:ffff8ad782c55600
juin 09 02:35:02 hermes kernel: RDX: 0000000002040001 RSI:00000000ffffffe4RDI:ffffffffc11dcac0
juin 09 02:35:02 hermes kernel: RBP:ffff8ad783d7bc78R08: 0000000000000000 R09:ffffffffc0ec412a
juin 09 02:35:02 hermes kernel: R10:fffff4c487b11b40R11:ffff8ad86c46d460R12: 0000000000000000
juin 09 02:35:02 hermes kernel: R13:ffff8ad6a2573c00R14: 0000000000001252 R15: 0000000000000000
juin 09 02:35:02 hermes kernel: FS: 0000000000000000(0000) GS:ffff8adee617c000(0000) knlGS:0000000000000000
juin 09 02:35:02 hermes kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
juin 09 02:35:02 hermes kernel: CR2:00007ffd95039e68CR3:000000072ca2c005CR4:00000000003726f0
juin 09 02:35:02 hermes kernel: Call Trace:
juin 09 02:35:02 hermes kernel: <TASK>
juin 09 02:35:02 hermes kernel:flush_space+0x126/0x5a0[btrfs]
juin 09 02:35:02 hermes kernel: ?btrfs_reduce_alloc_profile+0x5a/0x1d0[btrfs]
juin 09 02:35:02 hermes kernel: ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
juin 09 02:35:02 hermes kernel:do_async_reclaim_metadata_space+0x7d/0x180[btrfs]
juin 09 02:35:02 hermes kernel:btrfs_async_reclaim_metadata_space+0x54/0x70[btrfs]
juin 09 02:35:02 hermes kernel:process_one_work+0x19d/0x3a0
juin 09 02:35:02 hermes kernel:worker_thread+0x1af/0x320
juin 09 02:35:02 hermes kernel: ?__pfx_worker_thread+0x10/0x10
juin 09 02:35:02 hermes kernel:kthread+0xe3/0x120
juin 09 02:35:02 hermes kernel: ?__pfx_kthread+0x10/0x10
juin 09 02:35:02 hermes kernel:ret_from_fork+0x2c7/0x350
juin 09 02:35:02 hermes kernel: ?__pfx_kthread+0x10/0x10
juin 09 02:35:02 hermes kernel:ret_from_fork_asm+0x1a/0x30
juin 09 02:35:02 hermes kernel: </TASK>
juin 09 02:35:02 hermes kernel: ---[ end trace 0000000000000000 ]---
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): dumping space info:
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):space_infoDATA (sub-group id 0) has 17422733312 free, is full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):space_infototal=70262980608, used=52839989248, pinned=0, reserved=0, may_use=192512, readonly=65536 zone_unusable=0 juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):space_infoMETADATA (sub-group id 0) has 442368 free, is full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):space_infototal=3221225472, used=2155544576, pinned=38010880, reserved=10387456, may_use=576634880, readonly=440205312 zone_unusable=0 juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):space_infoSYSTEM (sub-group id 0) has 33538048 free, is not full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):space_infototal=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):global_block_rsv: size 190529536 reserved 190480384
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):trans_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):chunk_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):remap_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):delayed_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A):delayed_refs_rsv: size 1270611968 reserved 368197632
juin 09 02:35:02 hermes kernel: BTRFS: error (device sda3 state A) in btrfs_run_delayed_refs:2247: errno=-28 No space left
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): forced readonly
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): 1 enospc errors during balance
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): balance: ended with status: -30Linux (none) 7.0.10+deb14-amd64 #1 SMP
PREEMPT_DYNAMICDebian 7.0.10-1 (2026-05-27)x86_64GNU/Linuxbtrfs-progs v7.0
-EXPERIMENTAL-INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin#
/etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices# that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> LABEL=ROOT / btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2 0 1 UUID=9C79-15CA /boot/efi vfat umask=0077 0 1 LABEL=HDDHOME /home btrfs defaults,compress=zstd,noatime,space_cache=v2 0 2 #LABEL=SSDHOME /ssdhome btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2 0 2 LABEL=SSDHOME /var btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2,subvol=@var 0 2 UUID=EAB2-8BE4 /boot/efi-old vfat umask=0077 0 1btrfs fi usage
/var/@var/
Overall:
Device size: 68.47GiB
Device allocated: 68.47GiB
Device unallocated: 1.00MiB
Device missing: 0.00B
Device slack: 0.00B
Used: 51.21GiB
Free (estimated): 16.23GiB (min: 16.23GiB) Free (statfs, df): 16.23GiB
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 95.19MiB (used: 0.00B) Multiple profiles: noData,single: Size:65.44GiB, Used:49.21GiB (75.20%) /dev/sda3 65.44GiB
Metadata,single: Size:3.00GiB, Used:2.00GiB (66.51%) /dev/sda3 3.00GiB
System,single: Size:32.00MiB, Used:16.00KiB (0.05%) /dev/sda3 32.00MiB
Unallocated:
/dev/sda3 1.00MiBbtrfs balance status
/var/@var/
Balance on '/var/@var/' is paused
0 out of about 0 chunks balanced (0 considered), -nan% leftbtrfs balance cancel
/var/@var/
ERROR: balance cancel on '/var/@var/' failed: Read-only file systemThank you for your help,
Alban
Try:
mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var btrfs balance start -dlimit=1 /var
If the first command works but the second command does not, you could insert a btrfs device add command to add a partition. I'm not such a big fan of adding USB sticks or zram devices because if anything goes wrong, the file system ends up broken.
This is a bit of a heavy hammer to fix this problem but it might actually be faster than troubleshooting Btrfs enospc:
It should still be possible to make the troubled Btrfs a read-only seed device using btrfs-tune; then mount it ro; use btrfs dev add using a device that's equal to or bigger; remount rw; then btrfs dev rem the 1st device, i.e. the ro seed. This will replicate/clone the file system from device 1 to device 2. And it'll take a while. But in the end you'll have a balanced file system on device 2.
It isn't strictly necessary for device 2 to be equal to or bigger. It is possible to add a smaller device, but before you remove the 1st device, you wlil need to delete files. The bigger files you choose to delete, the fewer you will need to delete to get to the underprovisioned 2nd device size. Then remove the 1st device. Now only the files that have not been deleted will have extents replicated to the 2nd device.
At the very least the seed-device is not being modified and shouldn't get any worse than it is.
Chris Murphy
Le 09/06/2026 à 17:25, Chris Murphy a écrit :
Quoted 4 lines
Try:
mount -o skip_balance,nospace_cache,enospc_debug
/dev/sdb3/var btrfs balance start -dlimit=1 /var
Thanks for the suggestions. Unfortunately, the bypass did not work.
# mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var
[117052.507168] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (2004)
[117052.526475] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[117052.536146] BTRFS info (device sdb3): using crc32c checksum algorithm
[117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree
[117052.550097] BTRFS error (device sdb3): open_ctree failed: -22
mount: /var: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.I believe this issue is blocking me:
[117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree
So I tried with enospc_debug without nospace_cache, to get debug infos:
# mount -o skip_balance,enospc_debug /dev/sdb3 /var
[117261.590047] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (2008)
[117261.612863] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[117261.622548] BTRFS info (device sdb3): using crc32c checksum algorithm
[117261.662415] BTRFS info (device sdb3): left=0, need=393216, flags=36
[117261.669328] BTRFS info (device sdb3): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[117261.678792] BTRFS info (device sdb3): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[117261.692344] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190382080
[117261.701131] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0
[117261.708349] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0
[117261.715571] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0
[117261.722782] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0
[117261.730154] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0
[117261.737462] BTRFS error (device sdb3): allocation failed flags 36, wanted 16384 tree-log 0, relocation: 0
[117261.747497] BTRFS info (device sdb3): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117261.757371] BTRFS info (device sdb3): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117261.772172] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190365696
[117261.780844] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0
[117261.787995] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0
[117261.795127] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0
[117261.802251] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0
[117261.809551] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0
[117261.816785] BTRFS info (device sdb3): block group 55856594944 has 1073741824 bytes, 1007976448 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (65765376
bytes available) [readonly]
[117261.834327] BTRFS info (device sdb3): block group has cluster?: no
[117261.840947] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes
[117261.849500] BTRFS info (device sdb3): block group 101368987648 has 1073741824 bytes, 500957184 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (57278464
0 bytes available) [readonly]
[117261.867163] BTRFS info (device sdb3): block group has cluster?: no
[117261.873784] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.882313] BTRFS info (device sdb3): block group 102442729472 has 1073741824 bytes, 633536512 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (44020531
2 bytes available) [readonly]
[117261.899994] BTRFS info (device sdb3): block group has cluster?: no
[117261.906630] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes
[117261.915176] BTRFS info (device sdb3): 1078755328 bytes available across all block groups
[117261.923750] BTRFS warning (device sdb3): Skipping commit of aborted transaction.
[117261.931608] ------------[ cut here ]------------
[117261.936687] BTRFS: Transaction aborted (error -28)
[117261.941950] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#6: mount/2008
[117261.952905] Modules linked in: ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod auto
fs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper cec sd_mod rc_core drm_cli
ent_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 video realtek crc_itu_t usbcore scsi_
mod fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs
[117262.004021] CPU: 6 UID: 0 PID: 2008 Comm: mount Tainted: G W 7.0.10+deb14-amd64 #1 PREEMPT(lazy) Debian 7.0.10-1
[117262.016343] Tainted: [W]=WARN
[117262.019853] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
[117262.027348] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs]
[117262.034117] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9
f8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00
[117262.053426] RSP: 0018:ffffce9fc8b57418 EFLAGS: 00010292
[117262.059233] RAX: 0000000000000002 RBX: ffff8ac4d181a738 RCX: 0000000000000003
[117262.066946] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080
[117262.074658] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fc8b57200
[117262.082364] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac545885000
[117262.090075] R13: ffff8ac4c242ea00 R14: ffff8ac5458854c0 R15: ffff8ac4cff01b80
[117262.097779] FS: 00007f22c28cf940(0000) GS:ffff8acc3a4fc000(0000) knlGS:0000000000000000
[117262.106429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[117262.112740] CR2: 00007f22c2cdfdd0 CR3: 00000001867ac001 CR4: 00000000003706f0
[117262.120434] Call Trace:
[117262.123451] <TASK>
[117262.126121] ? start_transaction+0x228/0x880 [btrfs]
[117262.131676] flush_space+0x3c3/0x5a0 [btrfs]
[117262.136574] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[117262.142855] ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
[117262.149597] priority_reclaim_metadata_space+0x96/0x130 [btrfs]
[117262.156132] handle_reserve_ticket+0x5b/0x290 [btrfs]
[117262.161800] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[117262.168076] reserve_bytes+0x268/0x4c0 [btrfs] [117262.173127] btrfs_reserve_metadata_bytes+0x1a/0xe0 [btrfs]
[117262.179306] btrfs_block_rsv_refill+0x60/0x90 [btrfs]
[117262.184965] evict_refill_and_join+0x47/0xe0 [btrfs]
[117262.190531] btrfs_evict_inode+0x332/0x400 [btrfs]
[117262.195922] evict+0xfd/0x270
[117262.199441] btrfs_orphan_cleanup+0x21f/0x390 [btrfs]
[117262.205085] btrfs_start_pre_rw_mount+0x2a0/0x6c0 [btrfs]
[117262.211071] open_ctree+0x12fb/0x1730 [btrfs]
[117262.216028] btrfs_get_tree.cold+0xb/0xb6 [btrfs]
[117262.221324] ? security_inode_permission+0x4a/0x110
[117262.226747] vfs_get_tree+0x29/0xd0
[117262.230776] vfs_cmd_create+0x57/0xd0
[117262.234947] __do_sys_fsconfig+0x2fc/0x5e0
[117262.239581] do_syscall_64+0xcd/0x15f0
[117262.243873] ? __fs_parse+0x68/0x1c0
[117262.247989] ? aa_file_perm+0x18a/0x550
[117262.252368] ? btrfs_parse_param+0x4f/0x960 [btrfs]
[117262.257818] ? apparmor_file_permission+0x44/0xb0
[117262.263052] ? fscontext_read+0x171/0x190
[117262.267594] ? rw_verify_area+0x56/0x180
[117262.272039] ? vfs_read+0xbf/0x390
[117262.275964] ? __fs_parse+0x68/0x1c0
[117262.280063] ? aa_file_perm+0x18a/0x550
[117262.284422] ? btrfs_parse_param+0x4f/0x960 [btrfs]
[117262.289858] ? apparmor_file_permission+0x44/0xb0
[117262.295082] ? fscontext_read+0x171/0x190
[117262.299605] ? rw_verify_area+0x56/0x180
[117262.304036] ? vfs_read+0xbf/0x390
[117262.307943] ? do_syscall_64+0x10a/0x15f0
[117262.312451] ? from_kgid_munged+0x17/0x30
[117262.316955] ? __do_sys_getgid+0x27/0x30
[117262.321369] ? do_syscall_64+0x10a/0x15f0
[117262.325856] ? from_kgid_munged+0x17/0x30
[117262.330340] ? ksys_read+0x73/0xf0
[117262.334213] ? do_syscall_64+0x10a/0x15f0
[117262.338694] ? irqentry_exit+0x78/0x680
[117262.343000] ? exc_page_fault+0x82/0x1d0
[117262.347384] ? __irq_exit_rcu+0x3d/0xe0 [117262.351683] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[117262.357197] RIP: 0033:0x7f22c2d0eb4a
[117262.361218] Code: 73 01 c3 48 8b 0d c6 32 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff
73 01 c3 48 8b 0d 96 32 0d 00 f7 d8 64 89 01 48
[117262.380431] RSP: 002b:00007ffd5d547f68 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
[117262.388464] RAX: ffffffffffffffda RBX: 0000557471e4da50 RCX: 00007f22c2d0eb4a
[117262.396057] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
[117262.403650] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
[117262.411240] R10: 0000000000000000 R11: 0000000000000246 R12: 0000557471e4ec40
[117262.418831] R13: 00007f22c2ea127c R14: 00007f22c2e9f460 R15: 00007f22c28cf7f0
[117262.426410] </TASK>
[117262.429024] ---[ end trace 0000000000000000 ]---
[117262.434074] BTRFS info (device sdb3 state A): dumping space info:
[117262.440595] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full
[117262.451184] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[117262.465659] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.476171] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.491598] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[117262.501477] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[117262.515439] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080
[117262.524731] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0
[117262.532538] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0
[117262.540337] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0
[117262.548136] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0
[117262.556110] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0
[117262.563998] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left
[117262.573453] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.584033] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.599563] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080
[117262.608943] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0
[117262.616846] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0
[117262.624749] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0
[117262.632643] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0
[117262.640715] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0
[117262.648696] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.659312] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.674875] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080
[117262.684287] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0
[117262.692225] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0
[117262.700155] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0
[117262.708086] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0
[117262.716191] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0
[117262.724260] BTRFS warning (device sdb3 state EA): could not allocate space for delete; will truncate on mount
[117262.734622] BTRFS error (device sdb3 state EA): Error removing orphan entry, stopping orphan cleanup
[117262.744220] BTRFS error (device sdb3 state EA): could not do orphan cleanup -28
[117262.752021] BTRFS error (device sdb3 state EA): commit super block returned -30
[117262.760739] BTRFS error (device sdb3 state EA): open_ctree failed: -28
mount: /var: fsconfig() failed: No space left on device.
dmesg(1) may have more information after failed mount system call.Quoted 3 lines
If the first command works but the second command does not, you could insert a
btrfs device addcommand to add a partition. I'm not such a big fan of adding USB sticks or zram devices because if anything goes wrong, the file system ends up broken.This is a bit of a heavy hammer to fix this problem but it might actually be faster than troubleshooting Btrfs enospc:
I am all for troubleshooting this enospc issue because it is the first time I cannot even add a device to fix the enospc... in fact an hour before this issue I had an enospc issue this time with the root partition that had very few space left, and adding a device worked well and it was pretty easy to fix. But here I have gigabytes of unused data and metadata, I am puzzled.
I was switching my SSD btrfs partitions (I plan to merge them all but for historical reason they are split as separate btrfs partitions instead of subvolumes of a single btrfs partition) from single to DUP, the new default.
I believe the main issue here is that the conversion attempted to use until the last megabyte of metadata before errorring out, preventing any simple rescue operation afterwards, and even preventing from balancing data to free chunks for metadata.
It would have been fine for the conversion to error out and let me balance data then resume the conversion. The main issue is that the conversion used so much metadata space that I cannot even mount the btrfs partition to balance its data space.
Maybe the metadata conversion should guard/ straight error out when there is not enough space in metadata (here 3GB to duplicate 1,59GB does not fit) and there is no way to get a new free chunk for metadata and tell to free chunks first ?
Quoted 6 lines
It should still be possible to make the troubled Btrfs a read-only seed device using btrfs-tune; then mount it ro; use
btrfs dev addusing a device that's equal to or bigger; remount rw; thenbtrfs dev remthe 1st device, i.e. the ro seed. This will replicate/clone the file system from device 1 to device 2. And it'll take a while. But in the end you'll have a balanced file system on device 2.It isn't strictly necessary for device 2 to be equal to or bigger. It is possible to add a smaller device, but before you remove the 1st device, you wlil need to delete files. The bigger files you choose to delete, the fewer you will need to delete to get to the underprovisioned 2nd device size. Then remove the 1st device. Now only the files that have not been deleted will have extents replicated to the 2nd device.
At the very least the seed-device is not being modified and shouldn't get any worse than it is. Chris Murphy
I don't know yet how to proceed with such a read-only seed option, I will look into it. But is there other flags or tune I could do that could let me cancel the pause operation, the balance the data space, then retry the conversion ?
Alban
Here is the debug dump of this partition sdb3_var_metadata-only-debug.img at https://drive.google.com/file/d/1OzEZLaTUNpWsL1ohGKOMW4GiR9oQakdP/view?usp=drive_link done with "btrfs-image -c 9 -t 4 /dev/sdb3 /mnt/4/hermes-var-20260610-enospc-vanilla/sdb3_var_metadata-only-debug.img".
I tried to btrfs-resotre this partition but I get a lot of errors:
btrfs restore -v -m -x -i /dev/sdb3 /mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/
Skipping existing file /mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/@var/backups/group.bak If you wish to overwrite use -o
ERROR: zstd frame incomplete
ERROR: copying data for /mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/@var/backups/dpkg.arch.5.gz failed
ERROR: zstd frame incomplete
(...)
Alban
Le 10/06/2026 à 23:05, Alban Browaeys a écrit :
Quoted 212 lines
Le 09/06/2026 à 17:25, Chris Murphy a écrit :
Quoted 4 lines
Try:
mount -o skip_balance,nospace_cache,enospc_debug
/dev/sdb3/var btrfs balance start -dlimit=1 /varThanks for the suggestions. Unfortunately, the bypass did not work.
# mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var [117052.507168] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (2004) [117052.526475] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [117052.536146] BTRFS info (device sdb3): using crc32c checksum algorithm [117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree [117052.550097] BTRFS error (device sdb3): open_ctree failed: -22 mount: /var: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call.I believe this issue is blocking me:
[117052.543125] BTRFS error (device sdb3): cannot disable free-space-treeSo I tried with
enospc_debugwithoutnospace_cache, to get debug infos:# mount -o skip_balance,enospc_debug /dev/sdb3 /var [117261.590047] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (2008) [117261.612863] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [117261.622548] BTRFS info (device sdb3): using crc32c checksum algorithm [117261.662415] BTRFS info (device sdb3): left=0, need=393216, flags=36 [117261.669328] BTRFS info (device sdb3): space_info SYSTEM (sub-group id 0) has 0 free, is not full [117261.678792] BTRFS info (device sdb3): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0 [117261.692344] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190382080 [117261.701131] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0 [117261.708349] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0 [117261.715571] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0 [117261.722782] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0 [117261.730154] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0 [117261.737462] BTRFS error (device sdb3): allocation failed flags 36, wanted 16384 tree-log 0, relocation: 0 [117261.747497] BTRFS info (device sdb3): space_info METADATA (sub-group id 0) has -190382080 free, is full [117261.757371] BTRFS info (device sdb3): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 [117261.772172] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190365696 [117261.780844] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0 [117261.787995] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0 [117261.795127] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0 [117261.802251] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0 [117261.809551] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0 [117261.816785] BTRFS info (device sdb3): block group 55856594944 has 1073741824 bytes, 1007976448 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (65765376 bytes available) [readonly] [117261.834327] BTRFS info (device sdb3): block group has cluster?: no [117261.840947] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.849500] BTRFS info (device sdb3): block group 101368987648 has 1073741824 bytes, 500957184 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (57278464 0 bytes available) [readonly] [117261.867163] BTRFS info (device sdb3): block group has cluster?: no [117261.873784] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.882313] BTRFS info (device sdb3): block group 102442729472 has 1073741824 bytes, 633536512 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (44020531 2 bytes available) [readonly] [117261.899994] BTRFS info (device sdb3): block group has cluster?: no [117261.906630] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.915176] BTRFS info (device sdb3): 1078755328 bytes available across all block groups [117261.923750] BTRFS warning (device sdb3): Skipping commit of aborted transaction. [117261.931608] ------------[ cut here ]------------ [117261.936687] BTRFS: Transaction aborted (error -28) [117261.941950] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#6: mount/2008 [117261.952905] Modules linked in: ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod auto fs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper cec sd_mod rc_core drm_cli ent_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 video realtek crc_itu_t usbcore scsi_ mod fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs [117262.004021] CPU: 6 UID: 0 PID: 2008 Comm: mount Tainted: G W 7.0.10+deb14-amd64 #1 PREEMPT(lazy) Debian 7.0.10-1 [117262.016343] Tainted: [W]=WARN [117262.019853] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023 [117262.027348] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs] [117262.034117] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9 f8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00 [117262.053426] RSP: 0018:ffffce9fc8b57418 EFLAGS: 00010292 [117262.059233] RAX: 0000000000000002 RBX: ffff8ac4d181a738 RCX: 0000000000000003 [117262.066946] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080 [117262.074658] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fc8b57200 [117262.082364] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac545885000 [117262.090075] R13: ffff8ac4c242ea00 R14: ffff8ac5458854c0 R15: ffff8ac4cff01b80 [117262.097779] FS: 00007f22c28cf940(0000) GS:ffff8acc3a4fc000(0000) knlGS:0000000000000000 [117262.106429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [117262.112740] CR2: 00007f22c2cdfdd0 CR3: 00000001867ac001 CR4: 00000000003706f0 [117262.120434] Call Trace: [117262.123451] <TASK> [117262.126121] ? start_transaction+0x228/0x880 [btrfs] [117262.131676] flush_space+0x3c3/0x5a0 [btrfs] [117262.136574] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs] [117262.142855] ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs] [117262.149597] priority_reclaim_metadata_space+0x96/0x130 [btrfs] [117262.156132] handle_reserve_ticket+0x5b/0x290 [btrfs] [117262.161800] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs] [117262.168076] reserve_bytes+0x268/0x4c0 [btrfs] [117262.173127] btrfs_reserve_metadata_bytes+0x1a/0xe0 [btrfs] [117262.179306] btrfs_block_rsv_refill+0x60/0x90 [btrfs] [117262.184965] evict_refill_and_join+0x47/0xe0 [btrfs] [117262.190531] btrfs_evict_inode+0x332/0x400 [btrfs] [117262.195922] evict+0xfd/0x270 [117262.199441] btrfs_orphan_cleanup+0x21f/0x390 [btrfs] [117262.205085] btrfs_start_pre_rw_mount+0x2a0/0x6c0 [btrfs] [117262.211071] open_ctree+0x12fb/0x1730 [btrfs] [117262.216028] btrfs_get_tree.cold+0xb/0xb6 [btrfs] [117262.221324] ? security_inode_permission+0x4a/0x110 [117262.226747] vfs_get_tree+0x29/0xd0 [117262.230776] vfs_cmd_create+0x57/0xd0 [117262.234947] __do_sys_fsconfig+0x2fc/0x5e0 [117262.239581] do_syscall_64+0xcd/0x15f0 [117262.243873] ? __fs_parse+0x68/0x1c0 [117262.247989] ? aa_file_perm+0x18a/0x550 [117262.252368] ? btrfs_parse_param+0x4f/0x960 [btrfs] [117262.257818] ? apparmor_file_permission+0x44/0xb0 [117262.263052] ? fscontext_read+0x171/0x190 [117262.267594] ? rw_verify_area+0x56/0x180 [117262.272039] ? vfs_read+0xbf/0x390 [117262.275964] ? __fs_parse+0x68/0x1c0 [117262.280063] ? aa_file_perm+0x18a/0x550 [117262.284422] ? btrfs_parse_param+0x4f/0x960 [btrfs] [117262.289858] ? apparmor_file_permission+0x44/0xb0 [117262.295082] ? fscontext_read+0x171/0x190 [117262.299605] ? rw_verify_area+0x56/0x180 [117262.304036] ? vfs_read+0xbf/0x390 [117262.307943] ? do_syscall_64+0x10a/0x15f0 [117262.312451] ? from_kgid_munged+0x17/0x30 [117262.316955] ? __do_sys_getgid+0x27/0x30 [117262.321369] ? do_syscall_64+0x10a/0x15f0 [117262.325856] ? from_kgid_munged+0x17/0x30 [117262.330340] ? ksys_read+0x73/0xf0 [117262.334213] ? do_syscall_64+0x10a/0x15f0 [117262.338694] ? irqentry_exit+0x78/0x680 [117262.343000] ? exc_page_fault+0x82/0x1d0 [117262.347384] ? __irq_exit_rcu+0x3d/0xe0 [117262.351683] entry_SYSCALL_64_after_hwframe+0x76/0x7e [117262.357197] RIP: 0033:0x7f22c2d0eb4a [117262.361218] Code: 73 01 c3 48 8b 0d c6 32 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 96 32 0d 00 f7 d8 64 89 01 48 [117262.380431] RSP: 002b:00007ffd5d547f68 EFLAGS: 00000246 ORIG_RAX: 00000000000001af [117262.388464] RAX: ffffffffffffffda RBX: 0000557471e4da50 RCX: 00007f22c2d0eb4a [117262.396057] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003 [117262.403650] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000 [117262.411240] R10: 0000000000000000 R11: 0000000000000246 R12: 0000557471e4ec40 [117262.418831] R13: 00007f22c2ea127c R14: 00007f22c2e9f460 R15: 00007f22c28cf7f0 [117262.426410] </TASK> [117262.429024] ---[ end trace 0000000000000000 ]--- [117262.434074] BTRFS info (device sdb3 state A): dumping space info: [117262.440595] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full [117262.451184] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0 [117262.465659] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full [117262.476171] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 [117262.491598] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full [117262.501477] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0 [117262.515439] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080 [117262.524731] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0 [117262.532538] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0 [117262.540337] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0 [117262.548136] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0 [117262.556110] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0 [117262.563998] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left [117262.573453] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full [117262.584033] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 [117262.599563] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080 [117262.608943] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0 [117262.616846] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0 [117262.624749] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0 [117262.632643] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0 [117262.640715] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0 [117262.648696] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full [117262.659312] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 [117262.674875] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080 [117262.684287] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0 [117262.692225] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0 [117262.700155] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0 [117262.708086] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0 [117262.716191] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0 [117262.724260] BTRFS warning (device sdb3 state EA): could not allocate space for delete; will truncate on mount [117262.734622] BTRFS error (device sdb3 state EA): Error removing orphan entry, stopping orphan cleanup [117262.744220] BTRFS error (device sdb3 state EA): could not do orphan cleanup -28 [117262.752021] BTRFS error (device sdb3 state EA): commit super block returned -30 [117262.760739] BTRFS error (device sdb3 state EA): open_ctree failed: -28 mount: /var: fsconfig() failed: No space left on device. dmesg(1) may have more information after failed mount system call.Quoted 3 lines
If the first command works but the second command does not, you could insert a
btrfs device addcommand to add a partition. I'm not such a big fan of adding USB sticks or zram devices because if anything goes wrong, the file system ends up broken.This is a bit of a heavy hammer to fix this problem but it might actually be faster than troubleshooting Btrfs enospc:
I am all for troubleshooting this enospc issue because it is the first time I cannot even add a device to fix the enospc... in fact an hour before this issue I had an enospc issue this time with the root partition that had very few space left, and adding a device worked well and it was pretty easy to fix. But here I have gigabytes of unused data and metadata, I am puzzled.
I was switching my SSD btrfs partitions (I plan to merge them all but for historical reason they are split as separate btrfs partitions instead of subvolumes of a single btrfs partition) from single to DUP, the new default.
I believe the main issue here is that the conversion attempted to use until the last megabyte of metadata before errorring out, preventing any simple rescue operation afterwards, and even preventing from balancing data to free chunks for metadata.
It would have been fine for the conversion to error out and let me balance data then resume the conversion. The main issue is that the conversion used so much metadata space that I cannot even mount the btrfs partition to balance its data space.
Maybe the metadata conversion should guard/ straight error out when there is not enough space in metadata (here 3GB to duplicate 1,59GB does not fit) and there is no way to get a new free chunk for metadata and tell to free chunks first ?
Quoted 6 lines
It should still be possible to make the troubled Btrfs a read-only seed device using btrfs-tune; then mount it ro; use
btrfs dev addusing a device that's equal to or bigger; remount rw; thenbtrfs dev remthe 1st device, i.e. the ro seed. This will replicate/clone the file system from device 1 to device 2. And it'll take a while. But in the end you'll have a balanced file system on device 2.It isn't strictly necessary for device 2 to be equal to or bigger. It is possible to add a smaller device, but before you remove the 1st device, you wlil need to delete files. The bigger files you choose to delete, the fewer you will need to delete to get to the underprovisioned 2nd device size. Then remove the 1st device. Now only the files that have not been deleted will have extents replicated to the 2nd device.
At the very least the seed-device is not being modified and shouldn't get any worse than it is. Chris Murphy
I don't know yet how to proceed with such a read-only seed option, I will look into it. But is there other flags or tune I could do that could let me cancel the pause operation, the balance the data space, then retry the conversion ?
Alban
Hi,
I had space after the sdb3 btrfs /var partition, a 9GB swap partition. I deleted this swap partition and expanded the sdb3 btrfs size with parted But I don't seem to be able to expand the btrfs filesystem inside sdb3 if I cannot mount it read write, is it so? Or is it only due to my pending operation? Can I get this btrfs partition back?
I have an image of this partiton but it seems I will have the same issue with this image and will be unable to restore it.
Is there a way to copy the content of the partition mounted in ro in its current state, create a new partition and copy the content back to it?
There are docker subvolumes on it wich seems they can be recreated by docker later on (I hope so) so rsync might be an option but if the docker subvolume need to be backed up rsync is not able to backup them correctly. And btrfs restore gives me thousands of
Also should I expect this behavior to happen everytime I do a btrfs filessytem conversion without checking I have enough metadata space available beforehand or will this be prevented by some new code one day ?
# parted /dev/sdb GNU Parted 3.7 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) select /dev/sdb Using /dev/sdb (parted) p Model: ATA SanDisk SD9SB8W1 (scsi) Disk /dev/sdb: 128GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags:
Number Start End Size File system Name Flags 1 1049kB 903MB 902MB fat32 EFI system partition boot, hidden, esp 2 903MB 39.8GB 38.9GB btrfs root 3 43.2GB 117GB 73.5GB btrfs ssdhome 4 119GB 128GB 9197MB linux-swap(v1) SSDSWAP swap
(parted) rm 4
(parted) resizepart 3 100%
(parted) p
Model: ATA SanDisk SD9SB8W1 (scsi)
Disk /dev/sdb: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags 1 1049kB 903MB 902MB fat32 EFI system partition boot, hidden, esp 2 903MB 39.8GB 38.9GB btrfs root 3 43.2GB 128GB 84.8GB btrfs ssdhome
# mount /dev/sdb3 /mnt/3
[441290.450174] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (7976)
[441290.461454] BTRFS info (device sdb3):
first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[441290.471065] BTRFS info (device sdb3): using crc32c checksum algorithm [441290.518919] BTRFS warning (device sdb3): Skipping commit of aborted transaction.
[441290.527296] ------------[ cut here ]------------
[441290.532875] BTRFS: Transaction aborted (error -28)
[441290.538625] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#2: mount/7976 [441290.550177] Modules linked in: nls_ascii nls_cp437 vfat fat exfat ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx x
or raid6_pq raid1 raid0 md_mod autofs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_displa y_helper cec sd_mod rc_core drm_client_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 vi deo realtek crc_itu_t usbcore scsi_mod fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs
[441290.605209] CPU: 2 UID: 0 PID: 7976 Comm: mount Tainted: G W
7.0.10+deb14-amd64 #1 PREEMPT(lazy) Debian 7.0.10-1 [441290.617556] Tainted: [W]=WARN
[441290.621060] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023 [441290.628556] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs]
[441290.635306] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9 f8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00 [441290.654597] RSP: 0018:ffffce9fcc3fb6b8 EFLAGS: 00010292 [441290.660386] RAX: 0000000000000002 RBX: ffff8ac4c06a1c78 RCX: 0000000000000003 [441290.668074] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080 [441290.675771] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fcc3fb4a0 [441290.683464] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac4c584f000 [441290.691152] R13: ffff8ac4c7211600 R14: ffff8ac4c584f4c0 R15: ffff8ac545b71b80 [441290.698842] FS: 00007f16a636e940(0000) GS:ffff8acc3a3fc000(0000) knlGS:0000000000000000
[441290.707489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[441290.713789] CR2: 00007f71f5dcb19c CR3: 000000034a2fa002 CR4: 00000000003706f0 [441290.721478] Call Trace:
[441290.724485] <TASK>
[441290.727145] ? start_transaction+0x228/0x880 [btrfs]
[441290.732712] flush_space+0x3c3/0x5a0 [btrfs]
[441290.737591] ? __schedule+0x489/0x1010
[441290.741896] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[441290.748152] ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
[441290.754844] priority_reclaim_metadata_space+0x96/0x130 [btrfs] [441290.761360] handle_reserve_ticket+0x5b/0x290 [btrfs] [441290.767011] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[441290.773268] reserve_bytes+0x268/0x4c0 [btrfs]
[441290.778314] btrfs_reserve_metadata_bytes+0x1a/0xe0 [btrfs]
[441290.784484] btrfs_block_rsv_refill+0x60/0x90 [btrfs]
[441290.790127] evict_refill_and_join+0x47/0xe0
[btrfs]
[441290.795671] btrfs_evict_inode+0x332/0x400 [btrfs]
[441290.801048] evict+0xfd/0x270
[441290.804556] btrfs_orphan_cleanup+0x21f/0x390 [btrfs]
[441290.810191] btrfs_start_pre_rw_mount+0x2a0/0x6c0 [btrfs]
[441290.816159] open_ctree+0x12fb/0x1730 [btrfs]
[441290.821099] btrfs_get_tree.cold+0xb/0xb6 [btrfs]
[441290.826385] ? ksys_read+0x73/0xf0
[441290.830311] vfs_get_tree+0x29/0xd0
[441290.834324] vfs_cmd_create+0x57/0xd0
[441290.838474] __do_sys_fsconfig+0x2fc/0x5e0
[441290.843059] do_syscall_64+0xcd/0x15f0
[441290.847326] ? from_kuid_munged+0x13/0x30
[441290.851847] ? __do_sys_geteuid+0x27/0x30
[441290.856373] ? do_syscall_64+0x10a/0x15f0
[441290.860889] ? __x64_sys_mkdir+0x39/0x50 [441290.865317] ? do_syscall_64+0x10a/0x15f0 [441290.869841] ? __x64_sys_statx+0x73/0xd0 [441290.874261] ? do_syscall_64+0x10a/0x15f0
[441290.878766] ? irqentry_exit+0x78/0x680
[441290.883089] ? exc_page_fault+0x82/0x1d0
[441290.887504] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[441290.893048] RIP: 0033:0x7f16a67adb4a
[441290.897115] Code: 73 01 c3 48 8b 0d c6 32 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 96 32 0d 00 f7 d8 64 89 01 48
[441290.916373] RSP: 002b:00007ffded6881b8 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
[441290.924455] RAX: ffffffffffffffda RBX: 00005603bb8eba50 RCX: 00007f16a67adb4a [441290.932099] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003 [441290.939743] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000 [441290.947387] R10: 0000000000000000 R11: 0000000000000246 R12: 00005603bb8ecb80 [441290.955041] R13: 00007f16a694027c R14: 00007f16a693e460 R15: 00007f16a636e7f0 [441290.962695] </TASK>
[441290.965396] ---[ end trace 0000000000000000 ]--- [441290.970507] BTRFS info (device sdb3 state A): dumping space info: [441290.977124] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full
[441290.987784] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0 [441291.002378] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full
[441291.012959] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[441291.028472] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[441291.038428] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0 [441291.052461] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080
[441291.061805] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0 [441291.069675] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0
[441291.077552] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0
[441291.085436] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0
[441291.093455] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0
[441291.101367] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left
[441291.110841] BTRFS warning (device sdb3 state EA): could not allocate space for delete; will truncate on mount
[441291.121191] BTRFS error (device sdb3 state EA): Error removing orphan entry, stopping orphan cleanup
[441291.130780] BTRFS error (device sdb3 state EA): could not do orphan cleanup -28 [441291.138588] BTRFS error (device sdb3 state EA): commit super block returned -30 [441291.147315] BTRFS error (device sdb3 state EA): open_ctree failed: -28
mount: /mnt/3: fsconfig() failed: No space left on device.
dmesg(1) may have more information after failed mount system call.
# mount -o ro,rescue=all /dev/sdb3 /var [442011.834443] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (8020) [442011.845803] BTRFS info (device sdb3 state S): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [442011.856089] BTRFS info (device sdb3 state S): using crc32c checksum algorithm [442011.897204] BTRFS info (device sdb3 state ECS): enabling ssd optimizations [442011.904566] BTRFS info (device sdb3 state ECS): disabling log replay at mount time [442011.912626] BTRFS info (device sdb3 state ECS): turning on async discard [442011.919811] BTRFS info (device sdb3 state ECS): enabling free space tree [442011.927008] BTRFS info (device sdb3 state ECS): ignoring bad roots [442011.933661] BTRFS info (device sdb3 state ECS): ignoring data csums [442011.940357] BTRFS info (device sdb3 state ECS): ignoring meta csums [442011.947051] BTRFS info (device sdb3 state ECS): ignoring unknown super block flags # ls /var @var # btrfs fi resize max /var ERROR: unable to start resize, another exclusive operation 'balance paused' in progress
Le 11/06/2026 à 23:19, Alban Browaeys a écrit :
Quoted 13 lines
Here is the debug dump of this partition sdb3_var_metadata-only- debug.img at https://drive.google.com/file/ d/1OzEZLaTUNpWsL1ohGKOMW4GiR9oQakdP/view?usp=drive_link done with "btrfs-image -c 9 -t 4
/dev/sdb3/mnt/4/hermes-var-20260610-enospc-vanilla/sdb3_var_metadata-only-debug.img".I tried to btrfs-resotre this partition but I get a lot of errors:
btrfs restore -v -m -x -i
/dev/sdb3/mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/Skipping existing file
/mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/@var/backups/group.bak If you wish to overwrite use -o
ERROR: zstd frame incomplete
ERROR: copying data for/mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/@var/backups/dpkg.arch.5.gz failed
ERROR: zstd frame incomplete
(...)
Quoted 208 lines
Alban
Le 10/06/2026 à 23:05, Alban Browaeys a écrit :
Quoted 205 lines
Le 09/06/2026 à 17:25, Chris Murphy a écrit :
Quoted 4 lines
Try:
mount -o skip_balance,nospace_cache,enospc_debug
/dev/sdb3/var btrfs balance start -dlimit=1 /varThanks for the suggestions. Unfortunately, the bypass did not work.
# mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var [117052.507168] BTRFS: device label SSDHOME devid 1 transid 44655875 / dev/sdb3 (8:19) scanned by mount (2004) [117052.526475] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [117052.536146] BTRFS info (device sdb3): using crc32c checksum algorithm [117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree [117052.550097] BTRFS error (device sdb3): open_ctree failed: -22 mount: /var: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call.I believe this issue is blocking me:
[117052.543125] BTRFS error (device sdb3): cannot disable free-space-treeSo I tried with
enospc_debugwithoutnospace_cache, to get debug infos:# mount -o skip_balance,enospc_debug /dev/sdb3 /var [117261.590047] BTRFS: device label SSDHOME devid 1 transid 44655875 / dev/sdb3 (8:19) scanned by mount (2008) [117261.612863] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [117261.622548] BTRFS info (device sdb3): using crc32c checksum algorithm [117261.662415] BTRFS info (device sdb3): left=0, need=393216, flags=36 [117261.669328] BTRFS info (device sdb3): space_info SYSTEM (sub-group id 0) has 0 free, is not full [117261.678792] BTRFS info (device sdb3): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0 [117261.692344] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190382080 [117261.701131] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0 [117261.708349] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0 [117261.715571] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0 [117261.722782] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0 [117261.730154] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0 [117261.737462] BTRFS error (device sdb3): allocation failed flags 36, wanted 16384 tree-log 0, relocation: 0 [117261.747497] BTRFS info (device sdb3): space_info METADATA (sub- group id 0) has -190382080 free, is full [117261.757371] BTRFS info (device sdb3): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 [117261.772172] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190365696 [117261.780844] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0 [117261.787995] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0 [117261.795127] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0 [117261.802251] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0 [117261.809551] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0 [117261.816785] BTRFS info (device sdb3): block group 55856594944 has 1073741824 bytes, 1007976448 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (65765376 bytes available) [readonly] [117261.834327] BTRFS info (device sdb3): block group has cluster?: no [117261.840947] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.849500] BTRFS info (device sdb3): block group 101368987648 has 1073741824 bytes, 500957184 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (57278464 0 bytes available) [readonly] [117261.867163] BTRFS info (device sdb3): block group has cluster?: no [117261.873784] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.882313] BTRFS info (device sdb3): block group 102442729472 has 1073741824 bytes, 633536512 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (44020531 2 bytes available) [readonly] [117261.899994] BTRFS info (device sdb3): block group has cluster?: no [117261.906630] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.915176] BTRFS info (device sdb3): 1078755328 bytes available across all block groups [117261.923750] BTRFS warning (device sdb3): Skipping commit of aborted transaction. [117261.931608] ------------[ cut here ]------------ [117261.936687] BTRFS: Transaction aborted (error -28) [117261.941950] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#6: mount/2008 [117261.952905] Modules linked in: ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod auto fs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper cec sd_mod rc_core drm_cli ent_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 video realtek crc_itu_t usbcore scsi_ mod fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs [117262.004021] CPU: 6 UID: 0 PID: 2008 Comm: mount Tainted: G W 7.0.10+deb14-amd64 #1 PREEMPT(lazy) Debian 7.0.10-1 [117262.016343] Tainted: [W]=WARN [117262.019853] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023 [117262.027348] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs] [117262.034117] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9 f8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00 [117262.053426] RSP: 0018:ffffce9fc8b57418 EFLAGS: 00010292 [117262.059233] RAX: 0000000000000002 RBX: ffff8ac4d181a738 RCX: 0000000000000003 [117262.066946] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080 [117262.074658] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fc8b57200 [117262.082364] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac545885000 [117262.090075] R13: ffff8ac4c242ea00 R14: ffff8ac5458854c0 R15: ffff8ac4cff01b80 [117262.097779] FS: 00007f22c28cf940(0000) GS:ffff8acc3a4fc000(0000) knlGS:0000000000000000 [117262.106429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [117262.112740] CR2: 00007f22c2cdfdd0 CR3: 00000001867ac001 CR4: 00000000003706f0 [117262.120434] Call Trace: [117262.123451] <TASK> [117262.126121] ? start_transaction+0x228/0x880 [btrfs] [117262.131676] flush_space+0x3c3/0x5a0 [btrfs] [117262.136574] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs] [117262.142855] ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs] [117262.149597] priority_reclaim_metadata_space+0x96/0x130 [btrfs] [117262.156132] handle_reserve_ticket+0x5b/0x290 [btrfs] [117262.161800] ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs] [117262.168076] reserve_bytes+0x268/0x4c0 [btrfs] [117262.173127] btrfs_reserve_metadata_bytes+0x1a/0xe0 [btrfs] [117262.179306] btrfs_block_rsv_refill+0x60/0x90 [btrfs] [117262.184965] evict_refill_and_join+0x47/0xe0 [btrfs] [117262.190531] btrfs_evict_inode+0x332/0x400 [btrfs] [117262.195922] evict+0xfd/0x270 [117262.199441] btrfs_orphan_cleanup+0x21f/0x390 [btrfs] [117262.205085] btrfs_start_pre_rw_mount+0x2a0/0x6c0 [btrfs] [117262.211071] open_ctree+0x12fb/0x1730 [btrfs] [117262.216028] btrfs_get_tree.cold+0xb/0xb6 [btrfs] [117262.221324] ? security_inode_permission+0x4a/0x110 [117262.226747] vfs_get_tree+0x29/0xd0 [117262.230776] vfs_cmd_create+0x57/0xd0 [117262.234947] __do_sys_fsconfig+0x2fc/0x5e0 [117262.239581] do_syscall_64+0xcd/0x15f0 [117262.243873] ? __fs_parse+0x68/0x1c0 [117262.247989] ? aa_file_perm+0x18a/0x550 [117262.252368] ? btrfs_parse_param+0x4f/0x960 [btrfs] [117262.257818] ? apparmor_file_permission+0x44/0xb0 [117262.263052] ? fscontext_read+0x171/0x190 [117262.267594] ? rw_verify_area+0x56/0x180 [117262.272039] ? vfs_read+0xbf/0x390 [117262.275964] ? __fs_parse+0x68/0x1c0 [117262.280063] ? aa_file_perm+0x18a/0x550 [117262.284422] ? btrfs_parse_param+0x4f/0x960 [btrfs] [117262.289858] ? apparmor_file_permission+0x44/0xb0 [117262.295082] ? fscontext_read+0x171/0x190 [117262.299605] ? rw_verify_area+0x56/0x180 [117262.304036] ? vfs_read+0xbf/0x390 [117262.307943] ? do_syscall_64+0x10a/0x15f0 [117262.312451] ? from_kgid_munged+0x17/0x30 [117262.316955] ? __do_sys_getgid+0x27/0x30 [117262.321369] ? do_syscall_64+0x10a/0x15f0 [117262.325856] ? from_kgid_munged+0x17/0x30 [117262.330340] ? ksys_read+0x73/0xf0 [117262.334213] ? do_syscall_64+0x10a/0x15f0 [117262.338694] ? irqentry_exit+0x78/0x680 [117262.343000] ? exc_page_fault+0x82/0x1d0 [117262.347384] ? __irq_exit_rcu+0x3d/0xe0 [117262.351683] entry_SYSCALL_64_after_hwframe+0x76/0x7e [117262.357197] RIP: 0033:0x7f22c2d0eb4a [117262.361218] Code: 73 01 c3 48 8b 0d c6 32 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 96 32 0d 00 f7 d8 64 89 01 48 [117262.380431] RSP: 002b:00007ffd5d547f68 EFLAGS: 00000246 ORIG_RAX: 00000000000001af [117262.388464] RAX: ffffffffffffffda RBX: 0000557471e4da50 RCX: 00007f22c2d0eb4a [117262.396057] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003 [117262.403650] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000 [117262.411240] R10: 0000000000000000 R11: 0000000000000246 R12: 0000557471e4ec40 [117262.418831] R13: 00007f22c2ea127c R14: 00007f22c2e9f460 R15: 00007f22c28cf7f0 [117262.426410] </TASK> [117262.429024] ---[ end trace 0000000000000000 ]--- [117262.434074] BTRFS info (device sdb3 state A): dumping space info: [117262.440595] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full [117262.451184] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0 [117262.465659] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full [117262.476171] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 [117262.491598] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full [117262.501477] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0 [117262.515439] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080 [117262.524731] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0 [117262.532538] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0 [117262.540337] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0 [117262.548136] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0 [117262.556110] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0 [117262.563998] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left [117262.573453] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full [117262.584033] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 [117262.599563] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080 [117262.608943] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0 [117262.616846] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0 [117262.624749] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0 [117262.632643] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0 [117262.640715] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0 [117262.648696] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full [117262.659312] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 [117262.674875] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080 [117262.684287] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0 [117262.692225] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0 [117262.700155] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0 [117262.708086] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0 [117262.716191] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0 [117262.724260] BTRFS warning (device sdb3 state EA): could not allocate space for delete; will truncate on mount [117262.734622] BTRFS error (device sdb3 state EA): Error removing orphan entry, stopping orphan cleanup [117262.744220] BTRFS error (device sdb3 state EA): could not do orphan cleanup -28 [117262.752021] BTRFS error (device sdb3 state EA): commit super block returned -30 [117262.760739] BTRFS error (device sdb3 state EA): open_ctree failed: -28 mount: /var: fsconfig() failed: No space left on device. dmesg(1) may have more information after failed mount system call.Quoted 3 lines
If the first command works but the second command does not, you could insert a
btrfs device addcommand to add a partition. I'm not such a big fan of adding USB sticks or zram devices because if anything goes wrong, the file system ends up broken.This is a bit of a heavy hammer to fix this problem but it might actually be faster than troubleshooting Btrfs enospc:
I am all for troubleshooting this enospc issue because it is the first time I cannot even add a device to fix the enospc... in fact an hour before this issue I had an enospc issue this time with the root partition that had very few space left, and adding a device worked well and it was pretty easy to fix. But here I have gigabytes of unused data and metadata, I am puzzled.
I was switching my SSD btrfs partitions (I plan to merge them all but for historical reason they are split as separate btrfs partitions instead of subvolumes of a single btrfs partition) from single to DUP, the new default.
I believe the main issue here is that the conversion attempted to use until the last megabyte of metadata before errorring out, preventing any simple rescue operation afterwards, and even preventing from balancing data to free chunks for metadata.
It would have been fine for the conversion to error out and let me balance data then resume the conversion. The main issue is that the conversion used so much metadata space that I cannot even mount the btrfs partition to balance its data space.
Maybe the metadata conversion should guard/ straight error out when there is not enough space in metadata (here 3GB to duplicate 1,59GB does not fit) and there is no way to get a new free chunk for metadata and tell to free chunks first ?
Quoted 6 lines
It should still be possible to make the troubled Btrfs a read-only seed device using btrfs-tune; then mount it ro; use
btrfs dev addusing a device that's equal to or bigger; remount rw; thenbtrfs dev remthe 1st device, i.e. the ro seed. This will replicate/clone the file system from device 1 to device 2. And it'll take a while. But in the end you'll have a balanced file system on device 2.It isn't strictly necessary for device 2 to be equal to or bigger. It is possible to add a smaller device, but before you remove the 1st device, you wlil need to delete files. The bigger files you choose to delete, the fewer you will need to delete to get to the underprovisioned 2nd device size. Then remove the 1st device. Now only the files that have not been deleted will have extents replicated to the 2nd device.
At the very least the seed-device is not being modified and shouldn't get any worse than it is. Chris Murphy
So with I tried with what I understood:
with a copy of the image I did of the partition: /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
# btrfstune -S 1 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: please finish/cancel the running replace/balance before running this command
I tried:
# btrfs check --clear-space-cache v2 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img Opening filesystem to check... Checking filesystem on /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img UUID: 13af326c-631f-482b-9c34-b59b4f100608 WARNING: --clear-space-cache option is deprecated, please use "btrfs rescue clear-space-cache" instead ERROR: please finish/cancel the running replace/balance before running this command
# btrfs balance cancel /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: not a directory: /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
# btrfs rescue clear-uuid-tree /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: please finish/cancel the running replace/balance before running this command
and without the seed flag:
# dd if=/dev/zero of=/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var_target.img bs=1M count=90000 90000+0 records in 90000+0 records out 94371840000 bytes (94 GB, 88 GiB) copied, 1340.52 s, 70.4 MB/s
# losetup -f --show /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img [475088.702700] loop0: detected capacity change from 0 to 166119424 /dev/loop0
# mkdir -p /mnt/seed_workspace
# mount -o ro,rescue=all /dev/loop0 /mnt/seed_workspace [475287.536430] BTRFS info: device /dev/loop0 (7:0) using temp-fsid 7d3aa685-b9cd-4cbc-87ed-c2d0288fdeba [475287.546680] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/loop0 (7:0) scanned by mount (8576) [475287.556933] BTRFS info (device loop0 state S): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [475287.567325] BTRFS info (device loop0 state S): using crc32c checksum algorithm [475288.554301] BTRFS info (device loop0 state ECS): enabling ssd optimizations [475288.562226] BTRFS info (device loop0 state ECS): disabling log replay at mount time [475288.570394] BTRFS info (device loop0 state ECS): turning on async discard [475288.577648] BTRFS info (device loop0 state ECS): enabling free space tree [475288.584915] BTRFS info (device loop0 state ECS): ignoring bad roots [475288.591652] BTRFS info (device loop0 state ECS): ignoring data csums [475288.598482] BTRFS info (device loop0 state ECS): ignoring meta csums [475288.605308] BTRFS info (device loop0 state ECS): ignoring unknown super block flags
# losetup -f --show /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var_target.img [475310.285452] loop1: detected capacity change from 0 to 184320000 /dev/loop1
# btrfs device add /dev/loop1 /mnt/seed_workspace Performing full device TRIM /dev/loop1 (87.89GiB) ... ERROR: error adding device '/dev/loop1': Read-only file system
Quoted 6 lines
Quoted 6 lines
I don't know yet how to proceed with such a read-only seed option, I will look into it. But is there other flags or tune I could do that could let me cancel the pause operation, the balance the data space, then retry the conversion ?
Alban
Alban
Quoted 7 lines · Alban Browaeys
Hi,
I had space after the sdb3 btrfs /var partition, a 9GB swap partition. I deleted this swap partition and expanded the sdb3 btrfs size with parted But I don't seem to be able to expand the btrfs filesystem inside sdb3 if I cannot mount it read write, is it so? Or is it only due to my pending operation?
If the file system goes read-only it cannot be modified at all, including resize.
Quoted 1 line
Can I get this btrfs partition back?
I think there's a bug. I do know know if the bug wrote confusion to disk, but either way the current kernel code can't handle it so it's a bug. One of your dmesg contained this
Quoted 1 line
[ 1084.779326] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full
Can you run 'btrfs check --readonly' on this file system?
My recommendation is to stop mounting it rw. Only mount it ro. The more it is changed the worse the problem is likely to get.
Quoted 2 lines
Is there a way to copy the content of the partition mounted in ro in its current state, create a new partition and copy the content back to it?
mount -o ro
Then use rsync -a or cp -a
I think that's safest.
The other option I was suggesting might make it possible to fix, but I've never been able to try it in a case like what you're experiencing so it's entirely untested and therefore risky.
btrfstune -S1
This *changes the file system* therefore it's a risk. But I think it's a minimal change to just the superblock to make the file system a read-only seed device. This is the only exception to the rule that you cannot add a device when a Btrfs is read-only. It is possible to add a device to a seed device. But the unintuitive part is how to make it read-write.
btrfstune -S1 $device1
mount $device1 /mnt
btrfs device add $device2 /mnt
mount -o remount,rw /mnt
This is now a two device Btrfs file system. The seed device remains read-only. The second device receives all the writes/changes as a feature of COW. In effect it's a kind of overlay.
What I'm suggesting is if you now do:
btrfs device remove $device1 /mnt
This tells the btrfs kernel code to replicate the contents of device1 onto device2. This command will not complete until the replication is complete, which could take hours (no idea, depends on how full the file system is).
$device2 size needs to be at least as much as the used amount for $device1.
Quoted 4 lines
There are docker subvolumes on it wich seems they can be recreated by docker later on (I hope so) so rsync might be an option but if the docker subvolume need to be backed up rsync is not able to backup them correctly.
I do not know to what degree the docker btrfs graph driver uses reflinks. Using rsync or cp might dramatically explode the amount of storage needed. It could be docker usage is part of the problem if the problem is at all related to bookend extents. I seem to recall that docker graph driver does result in substantial amounts of bookend extents.
Quoted 1 line
And btrfs restore gives me thousands of
Btrfs restore may not help. It's a scraping tool. It's designed to get data out at all costs, even permitting extraction of corrupt files. It won't prefer reflinks or snapshots.
Quoted 4 lines
Also should I expect this behavior to happen everytime I do a btrfs filessytem conversion without checking I have enough metadata space available beforehand or will this be prevented by some new code one day ?
This is always the hard part about file systems. Computers are ordinarily deterministic. But file systems are increasingly non-deterministic as they age. And once you hit the bug, the state of the file system has already changed making it important to stop making all changes to the file system in order to preserve that state for file system developers. The more we hammer on trying to fix it, it's like a crime scene being cleaned up or tampered with. It makes it impossible for the developers to understand what happened and therefore how to fix it so it doesn't happen again.
Signature · 1 line
Chris Murphy
Hi Chris, Hi All,
There is still a thing I wonder, reagarding the size of the used metadata, and knowing that small data are stored on the metadata partition, could it be possible to force btrfs to move these small data items to the data partition that has 16GB free?
I doubt I have 1.5GB of pure metadata for around 50GB of data, even if there are a lot of small files.
Le 15/06/2026 à 04:07, Chris Murphy a écrit :
Quoted 14 lines · Alban Browaeys
Quoted 7 lines
Hi,
I had space after the sdb3 btrfs /var partition, a 9GB swap partition. I deleted this swap partition and expanded the sdb3 btrfs size with parted But I don't seem to be able to expand the btrfs filesystem inside sdb3 if I cannot mount it read write, is it so? Or is it only due to my pending operation?
If the file system goes read-only it cannot be modified at all, including resize.
Quoted 8 lines
Quoted 1 line
Can I get this btrfs partition back?
I think there's a bug. I do know know if the bug wrote confusion to disk, but either way the current kernel code can't handle it so it's a bug. One of your dmesg contained this
Quoted 1 line
[ 1084.779326] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is fullCan you run 'btrfs check --readonly' on this file system?
# btrfs check --readonly /dev/sdb3 Opening filesystem to check... Checking filesystem on /dev/sdb3 UUID: 13af326c-631f-482b-9c34-b59b4f100608 [1/8] checking log skipped (none written) [2/8] checking root items [3/8] checking extents [4/8] checking free space tree [5/8] checking fs roots [6/8] checking only csums items (without verifying data) [7/8] checking root refs [8/8] checking quota groups skipped (not enabled on this FS) found 54982475776 bytes used, no error found total csum bytes: 50964248 total tree bytes: 2142486528 total fs tree bytes: 1951760384 total extent tree bytes: 97910784 btree space waste bytes: 569359506 file data blocks allocated: 105614794752 referenced 100030590976
Quoted 1 line
My recommendation is to stop mounting it rw. Only mount it ro. The more it is changed the worse the problem is likely to get.
So I did. I don't know if the previous attempt to mount it read write with default options that filed with ENOSPC did damages. But I have since refrained from attempting to mount it rw with any rescue option, I only mounted ro with the rescue options.
Quoted 11 lines
Quoted 4 lines
Is there a way to copy the content of the partition mounted in ro in its current state, create a new partition and copy the content back to it?
mount -o ro
Then use rsync -a or cp -a
I think that's safest.
Thank you, the doubt I had with rsync where about it flatting
subvolumes. I don't know a tool that does what rsync does without
flattening subvolumes. Thankfully the only subvolumes on this
partitions are /var/li/docker/btrfs that Gemini told me would be
recreated at startup by the docker dameon, so it told me to exclude
them from the rsync. For now the only reliable backup I am confident is
complete is the dd raw image... but it suffers from the same issue the
raw device partition has, ie that it is likely in a possibly
recoverable corrupt state (extends halfway in migration). Gemini told
me to try btrfs-restore but it gave me thousands of errors like:
"
ERROR: zstd frame incomplete
ERROR: copying data for /mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/@var/backups/dpkg.arch.5.gz failed "
Quoted 11 lines
The other option I was suggesting might make it possible to fix, but I've never been able to try it in a case like what you're experiencing so it's entirely untested and therefore risky.
btrfstune -S1
This *changes the file system* therefore it's a risk. But I think it's a minimal change to just the superblock to make the file system a read-only seed device. This is the only exception to the rule that you cannot add a device when a Btrfs is read-only. It is possible to add a device to a seed device. But the unintuitive part is how to make it read-write.
btrfstune -S1 $device1
mount $device1 /mnt
btrfs device add $device2 /mnt
mount -o remount,rw /mnt
Sadly I tried it and reported about it in my previous email (but my bad I should have shortened the bug citation block in the middle which probably lead you to conclude there was only citation until the end of the email). The issue is I cannot mark the filesystem as seed if an operation is pending. (and as seen before I cannot cancel this operation it seems if I cannot mount the filesystem read write, dead lock). Here it is:
So with I tried with what I understood:
with a copy of the image I did of the partition:
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
# btrfstune -S 1 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: please finish/cancel the running replace/balance before running this command
I tried:
# btrfs check --clear-space-cache v2 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img Opening filesystem to check... Checking filesystem on /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img UUID: 13af326c-631f-482b-9c34-b59b4f100608 WARNING: --clear-space-cache option is deprecated, please use "btrfs rescue clear-space-cache" instead ERROR: please finish/cancel the running replace/balance before running this command
# btrfs balance cancel /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: not a directory: /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
# btrfs rescue clear-uuid-tree /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: please finish/cancel the running replace/balance before running this command
and without the seed flag:
# dd if=/dev/zero of=/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var_target.img bs=1M count=90000 90000+0 records in 90000+0 records out 94371840000 bytes (94 GB, 88 GiB) copied, 1340.52 s, 70.4 MB/s
# losetup -f --show /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img [475088.702700] loop0: detected capacity change from 0 to 166119424 /dev/loop0
# mkdir -p /mnt/seed_workspace
# mount -o ro,rescue=all /dev/loop0 /mnt/seed_workspace [475287.536430] BTRFS info: device /dev/loop0 (7:0) using temp-fsid 7d3aa685-b9cd-4cbc-87ed-c2d0288fdeba [475287.546680] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/loop0 (7:0) scanned by mount (8576) [475287.556933] BTRFS info (device loop0 state S): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [475287.567325] BTRFS info (device loop0 state S): using crc32c checksum algorithm [475288.554301] BTRFS info (device loop0 state ECS): enabling ssd optimizations [475288.562226] BTRFS info (device loop0 state ECS): disabling log replay at mount time [475288.570394] BTRFS info (device loop0 state ECS): turning on async discard [475288.577648] BTRFS info (device loop0 state ECS): enabling free space tree [475288.584915] BTRFS info (device loop0 state ECS): ignoring bad roots [475288.591652] BTRFS info (device loop0 state ECS): ignoring data csums [475288.598482] BTRFS info (device loop0 state ECS): ignoring meta csums [475288.605308] BTRFS info (device loop0 state ECS): ignoring unknown super block flags
# losetup -f --show /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var_target.img [475310.285452] loop1: detected capacity change from 0 to 184320000 /dev/loop1
# btrfs device add /dev/loop1 /mnt/seed_workspace Performing full device TRIM /dev/loop1 (87.89GiB) ... [475344.936908] BTRFS error (device loop0 state ECS): device add not supported on cloned temp-fsid mount ERROR: error adding device '/dev/loop1': Invalid argument
Quoted 15 lines
This is now a two device Btrfs file system. The seed device remains read-only. The second device receives all the writes/changes as a feature of COW. In effect it's a kind of overlay.
What I'm suggesting is if you now do:
btrfs device remove $device1 /mnt
This tells the btrfs kernel code to replicate the contents of device1 onto device2. This command will not complete until the replication is complete, which could take hours (no idea, depends on how full the file system is).
$device2 size needs to be at least as much as the used amount for $device1.
Quoted 4 lines
There are docker subvolumes on it wich seems they can be recreated by docker later on (I hope so) so rsync might be an option but if the docker subvolume need to be backed up rsync is not able to backup them correctly.
Quoted 6 lines
I do not know to what degree the docker btrfs graph driver uses reflinks. Using rsync or cp might dramatically explode the amount of storage needed. It could be docker usage is part of the problem if the problem is at all related to bookend extents. I seem to recall that docker graph driver does result in substantial amounts of bookend extents.
Quoted 1 line
And btrfs restore gives me thousands of
Btrfs restore may not help. It's a scraping tool. It's designed to get data out at all costs, even permitting extraction of corrupt files. It won't prefer reflinks or snapshots.
Ok. Though I believe that if btrfs-restore find incomplete zstd frame it might be that rsync will not be able to copy everything from the ro half converted btrfs partition?
Quoted 6 lines
Quoted 4 lines
Also should I expect this behavior to happen everytime I do a btrfs filessytem conversion without checking I have enough metadata space available beforehand or will this be prevented by some new code one day ?
This is always the hard part about file systems. Computers are ordinarily deterministic. But file systems are increasingly non-deterministic as they age. And once you hit the bug, the state of the file system has already changed making it important to stop making all changes to the file system in order to preserve that state for file system developers. The more we hammer on trying to fix it, it's like a crime scene being cleaned up or tampered with. It makes it impossible for the developers to understand what happened and therefore how to fix it so it doesn't happen again.
Sure. Out of plain "mount /dev/sdb3 /var" I haven't done any rw mount attempts. I hope these attempts did not make the issue worst.
Alban
Quoted 7 lines · Alban Browaeys
Hi Chris, Hi All,
There is still a thing I wonder, reagarding the size of the used metadata, and knowing that small data are stored on the metadata partition, could it be possible to force btrfs to move these small data items to the data partition that has 16GB free?
No. By default Btrfs has separate block group types so it's not possible for Btrfs to put metadata into the unused space in data block groups or vice versa. That is permitted with mixed block group type.
Quoted 12 lines
# btrfs check --readonly /dev/sdb3 Opening filesystem to check... Checking filesystem on /dev/sdb3 UUID: 13af326c-631f-482b-9c34-b59b4f100608 [1/8] checking log skipped (none written) [2/8] checking root items [3/8] checking extents [4/8] checking free space tree [5/8] checking fs roots [6/8] checking only csums items (without verifying data) [7/8] checking root refs [8/8] checking quota groups skipped (not enabled on this FS)
Quoted 8 lines
found 54982475776 bytes used, no error found total csum bytes: 50964248 total tree bytes: 2142486528 total fs tree bytes: 1951760384 total extent tree bytes: 97910784 btree space waste bytes: 569359506 file data blocks allocated: 105614794752 referenced 100030590976
OK so the file system is consistent. And there's no log tree.
For some reason skip_balance is not able to abort the balance/convert before the file system runs into the problem and goes read-only. Read-only prevents us from doing everything else.
Clearly the file system wants to allocate at least one metadata block group but cannot, there's no unallocated space on the device and no way to resize or add more space.
At the moment it appears to be stuck. So it's a bug somewhere.
Quoted 5 lines
Here it is:
So with I tried with what I understood:
with a copy of the image I did of the partition:
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
Quoted 7 lines
# btrfstune -S 1 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: please finish/cancel the running replace/balance before running this commandI tried:
# btrfs check --clear-space-cache v2
Yeah I advise not removing either version of space cache until the balance is cancelled. But we can't cancel the balance before the file system goes read-only which mean the balance cannot be cancelled. This is unrelated to the free space tree.
But yeah too bad the seed flag cannot be set on such a file system.
Quoted 15 lines
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.imgOpening filesystem to check...
Checking filesystem on/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
UUID: 13af326c-631f-482b-9c34-b59b4f100608
WARNING: --clear-space-cache option is deprecated, please use "btrfs rescue clear-space-cache" instead
ERROR: please finish/cancel the running replace/balance before running this command# btrfs balance cancel /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: not a directory: /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
Quoted 4 lines
# btrfs rescue clear-uuid-tree /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img ERROR: please finish/cancel the running replace/balance before running this command
Clearing trees while a balance is in progress is not recommended. So it's a good thing the tool doesn't permit it.
Quoted 3 lines
Ok. Though I believe that if btrfs-restore find incomplete zstd frame it might be that rsync will not be able to copy everything from the ro half converted btrfs partition?
I don't know what the btrfs restore complaint about zstd frames is about. That makes me think there is corrupt data blocks, and the block can't be decompressed by zstd.
You could try:
mount -o ro $dev1 /mnt
btrfs scrub start -Brd /mnt
That will do a read-only scrub on the read-only file system. It can't fix anything but it will show if there's corrupted files.
In any case, the file system does mount ro normally. You can reliably get important information out. The most portable backup/restore for podman/docker/mobi is tar - I know they have internal backup commands that use tar but I do not know what options are actually used and therefore how to make a tar backup of your containers rather than relying on rsync - it's not my area. But I would investigate a more generic way of getting important containers out of the broken file system so you can preserve them no matter the file system or graph driver you decide to use.
Signature · 1 line
Chris Murphy
Have you seen I posted a link to a btrfs debug image in a previous email in this thread?
"
partition sdb3_var_metadata-only-debug.img at https://drive.google.com/file/d/1OzEZLaTUNpWsL1ohGKOMW4GiR9oQakdP/view?usp=drive_link done with "btrfs-image -c 9 -t 4 /dev/sdb3 /mnt/4/hermes-var-20260610-enospc-vanilla/sdb3_var_metadata-only-debug.img". "
I made an image and a btrfs debug image, is there any use for me to keep the broken on disk partition? I believe I cannot expect to be able to recover this on disk partition in a week and thus should delete it and recreate it if I want the pc back? And keep the disk partition image for debugging.
Le 15/06/2026 à 22:00, Chris Murphy a écrit :
Quoted 37 lines · Alban Browaeys
Quoted 13 lines
# btrfs check --readonly /dev/sdb3 Opening filesystem to check... Checking filesystem on /dev/sdb3 UUID: 13af326c-631f-482b-9c34-b59b4f100608 [1/8] checking log skipped (none written) [2/8] checking root items [3/8] checking extents [4/8] checking free space tree [5/8] checking fs roots [6/8] checking only csums items (without verifying data) [7/8] checking root refs [8/8] checking quota groups skipped (not enabled on this FS)Quoted 8 lines
found 54982475776 bytes used, no error found total csum bytes: 50964248 total tree bytes: 2142486528 total fs tree bytes: 1951760384 total extent tree bytes: 97910784 btree space waste bytes: 569359506 file data blocks allocated: 105614794752 referenced 100030590976OK so the file system is consistent. And there's no log tree.
For some reason
skip_balanceis not able to abort the balance/convert before the file system runs into the problem and goes read-only. Read-only prevents us from doing everything else.Clearly the file system wants to allocate at least one metadata block group but cannot, there's no unallocated space on the device and no way to resize or add more space.
At the moment it appears to be stuck. So it's a bug somewhere.
Quoted 5 lines
Quoted 3 lines
Ok. Though I believe that if btrfs-restore find incomplete zstd frame it might be that rsync will not be able to copy everything from the ro half converted btrfs partition?
I don't know what the btrfs restore complaint about zstd frames is about. That makes me think there is corrupt data blocks, and the block can't be decompressed by zstd.
Could it be half transfered data during the conversion ?
Quoted 4 lines
You could try:
mount -o ro $dev1 /mnt
btrfs scrub start -Brd /mnt
# mount -o ro /dev/sdb3 /var [820430.153273] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (13961) [820430.175471] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608 [820430.185087] BTRFS info (device sdb3): using crc32c checksum algorithm [820430.221645] BTRFS info (device sdb3): enabling ssd optimizations [820430.228181] BTRFS info (device sdb3): turning on async discard [820430.234575] BTRFS info (device sdb3): enabling free space tree # btrfs scrub start -Brd /var WARNING: cannot create scrub data file, mkdir /v[820447.453110] BTRFS info (device sdb3): scrub: started on devid 1 ar/lib failed: Read-only file system, status recording disabled WARNING: failed to open the progress status socket at /var/lib/btrfs/scrub.progress.13af326c-631f-482b-9c34-b59b4f100608: No such file or directory, progress cannot be queried Starting scrub on devid 1 [820598.534072] BTRFS info (device sdb3): scrub: finished on devid 1 with status: 0
Scrub device /dev/sdb3 (id 1) done
Scrub started: Fri Jun 19 01:51:24 2026 Status: finished Duration: 0:02:31 Total to scrub: 51.21GiB Rate: 347.25MiB/s Error summary: no errors found
# btrfs fi df /var Data, single: total=65.44GiB, used=49.21GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=3.00GiB, used=2.00GiB GlobalReserve, single: total=181.56MiB, used=0.00B
Quoted 3 lines
That will do a read-only scrub on the read-only file system. It can't fix anything but it will show if there's corrupted files.
In any case, the file system does mount ro normally. You can reliably get important information out. The most portable backup/restore for podman/docker/mobi is tar - I know they have internal backup commands that use tar but I do not know what options are actually used and therefore how to make a tar backup of your containers rather than relying on rsync - it's not my area. But I would investigate a more generic way of getting important containers out of the broken file system so you can preserve them no matter the file system or graph driver you decide to use.
Alban