Skip to contentSkip to conversation

Btrfs ENOSPC / Stuck in RO with "exclusive operation balance paused in progress"

DISCUSSION Alban Browaeys 2 participants 10 messages Jun 9 – Jun 19 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hello,

On Debian Forky (testing), I tried to migrate my old SSD btrfs partition metadata from the previous default Single to the new DUP default.

The migration stopped halfway far /var and I am now stuck.

I am facing a severe ENOSPC deadlock it seems.

Symptoms:

- The filesystem conversion stopped midway and turned Read-Only during a balance operation.

- Attempting to remount or mount as 'rw' fails with ENOSPC.

- Adding a temporary loop device in RAM via 'btrfs device add' fails because the FS is read-only.

- Trying to run 'btrfs check --clear-space-cache v2' or cleanup tools at cold state fails with: "error: please finish/cancel the running replace/balance before running this command".

- 'btrfs balance cancel' cannot be run because mounting 'rw' is rejected, and it refuses to target the unmounted block device directly.

I can only boot into init=/bin/sh as of now.

Here are the outputs:

btrfs fi df /var
Data, single: total=65.44GiB, used=49.21GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=3.00GiB, used=1.59GiB
GlobalReserve, single: total=181.70MiB, used=0.00B
 prahal  ~  sudo btrfs balance start -mprofiles=single -mconvert=dup /var
ERROR: error during balancing '/var': Read-only file system
There may be more info in syslog - try dmesg | tail
uin 09 02:34:34 hermes kernel: BTRFS info (device sda3): balance: start -mconvert=dup,profiles=single -sconvert=dup,profiles=single
juin 09 02:34:34 hermes kernel: BTRFS info (device sda3): relocating block group 102442729472 flags metadata
juin 09 02:35:02 hermes kernel: BTRFS error (device sda3): failed to run delayed ref for logical 3976462336 num_bytes 8192 type 184 action 1 ref_mod 1: -28
juin 09 02:35:02 hermes kernel: ------------[ cut here ]------------
juin 09 02:35:02 hermes kernel: BTRFS: Transaction aborted (error -28)
juin 09 02:35:02 hermes kernel: WARNING: fs/btrfs/extent-tree.c:2247 at btrfs_run_delayed_refs+0x115/0x120 [btrfs], CPU#0: kworker/u32:11/1370737
juin 09 02:35:02 hermes kernel: Modules linked in: nilfs2 ufs qnx4 hfsplus hfs minix msdos jfs nls_ucs2_utils xfs ext4 mbcache jbd2 crc32c_cryptoapi cpuid exfat uinput rfcomm snd_seq_dummy snd_hrtimer snd_seq x> juin 09 02:35:02 hermes kernel:  snd_soc_avs nls_ascii snd_soc_hda_codec kvm_intel snd_hda_ext_core nls_cp437 snd_hda_codec vfat snd_hda_core fat kvm btrtl btbcm snd_intel_dspcfg snd_intel_sdw_acpi snd_hwdep gs> juin 09 02:35:02 hermes kernel:  hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper sd_mod cec rc_core iTCO_wdt drm_client_lib intel_pmc_bxt xhci_pci watchdog> juin 09 02:35:02 hermes kernel: CPU: 0 UID: 0 PID: 1370737 Comm: kworker/u32:11 Tainted: P           OE       7.0.9+deb14-amd64 #1 PREEMPT(lazy)  Debian 7.0.9-1
juin 09 02:35:02 hermes kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
juin 09 02:35:02 hermes kernel: Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
juin 09 02:35:02 hermes kernel: Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
juin 09 02:35:02 hermes kernel: RIP: 0010:btrfs_run_delayed_refs+0x117/0x120 [btrfs]
juin 09 02:35:02 hermes kernel: Code: 32 27 c1 48 89 ef e8 e8 54 0f 00 eb 07 e8 f1 22 ed d5 31 db 89 d8 5b 5d 41 5c 41 5d c3 cc cc cc cc 48 8d 3d 5b 7f 31 00 89 de <67> 48 0f b9 3a eb be 66 90 90 90 90 90 90 90> juin 09 02:35:02 hermes kernel: RSP: 0018:ffffd03f75b87d48 EFLAGS: 00010292
juin 09 02:35:02 hermes kernel: RAX: 0000000000000002 RBX: 00000000ffffffe4 RCX: ffff8ad782c55600
juin 09 02:35:02 hermes kernel: RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc11dcac0
juin 09 02:35:02 hermes kernel: RBP: ffff8ad783d7bc78 R08: 0000000000000000 R09: ffffffffc0ec412a
juin 09 02:35:02 hermes kernel: R10: fffff4c487b11b40 R11: ffff8ad86c46d460 R12: 0000000000000000
juin 09 02:35:02 hermes kernel: R13: ffff8ad6a2573c00 R14: 0000000000001252 R15: 0000000000000000
juin 09 02:35:02 hermes kernel: FS:  0000000000000000(0000) GS:ffff8adee617c000(0000) knlGS:0000000000000000
juin 09 02:35:02 hermes kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
juin 09 02:35:02 hermes kernel: CR2: 00007ffd95039e68 CR3: 000000072ca2c005 CR4: 00000000003726f0
juin 09 02:35:02 hermes kernel: Call Trace:
juin 09 02:35:02 hermes kernel:  <TASK>
juin 09 02:35:02 hermes kernel:  flush_space+0x126/0x5a0 [btrfs]
juin 09 02:35:02 hermes kernel:  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
juin 09 02:35:02 hermes kernel:  ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
juin 09 02:35:02 hermes kernel:  do_async_reclaim_metadata_space+0x7d/0x180 [btrfs]
juin 09 02:35:02 hermes kernel:  btrfs_async_reclaim_metadata_space+0x54/0x70 [btrfs]
juin 09 02:35:02 hermes kernel:  process_one_work+0x19d/0x3a0
juin 09 02:35:02 hermes kernel:  worker_thread+0x1af/0x320
juin 09 02:35:02 hermes kernel:  ? __pfx_worker_thread+0x10/0x10
juin 09 02:35:02 hermes kernel:  kthread+0xe3/0x120
juin 09 02:35:02 hermes kernel:  ? __pfx_kthread+0x10/0x10
juin 09 02:35:02 hermes kernel:  ret_from_fork+0x2c7/0x350
juin 09 02:35:02 hermes kernel:  ? __pfx_kthread+0x10/0x10
juin 09 02:35:02 hermes kernel:  ret_from_fork_asm+0x1a/0x30
juin 09 02:35:02 hermes kernel:  </TASK>
juin 09 02:35:02 hermes kernel: ---[ end trace 0000000000000000 ]---
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): dumping space info:
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info DATA (sub-group id 0) has 17422733312 free, is full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=192512, readonly=65536 zone_unusable=0 juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info METADATA (sub-group id 0) has 442368 free, is full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=3221225472, used=2155544576, pinned=38010880, reserved=10387456, may_use=576634880, readonly=440205312 zone_unusable=0 juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info SYSTEM (sub-group id 0) has 33538048 free, is not full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): global_block_rsv: size 190529536 reserved 190480384
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): trans_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): chunk_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): remap_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): delayed_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): delayed_refs_rsv: size 1270611968 reserved 368197632
juin 09 02:35:02 hermes kernel: BTRFS: error (device sda3 state A) in btrfs_run_delayed_refs:2247: errno=-28 No space left
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): forced readonly
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): 1 enospc errors during balance
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): balance: ended with status: -30

Linux (none) 7.0.10+deb14-amd64 #1 SMP PREEMPT_DYNAMIC Debian 7.0.10-1 (2026-05-27) x86_64 GNU/Linux

btrfs-progs v7.0
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices

# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
LABEL=ROOT 	/               btrfs   defaults,ssd,compress=zstd,noatime,space_cache=v2		0       1
UUID=9C79-15CA  /boot/efi       vfat    umask=0077					0       1
LABEL=HDDHOME 	/home           btrfs   defaults,compress=zstd,noatime,space_cache=v2			0       2
#LABEL=SSDHOME 	/ssdhome        btrfs   defaults,ssd,compress=zstd,noatime,space_cache=v2	        0       2
LABEL=SSDHOME 	/var            btrfs   defaults,ssd,compress=zstd,noatime,space_cache=v2,subvol=@var	0       2
UUID=EAB2-8BE4  /boot/efi-old       vfat    umask=0077					0       1

btrfs fi usage /var/@var/
Overall:

	Device size:		  68.47GiB
	Device allocated:		  68.47GiB
	Device unallocated:		   1.00MiB
	Device missing:		     0.00B
	Device slack:		     0.00B
	Used:			  51.21GiB
	Free (estimated):		  16.23GiB	(min: 16.23GiB)
	Free (statfs, df):		  16.23GiB
	Data ratio:			      1.00
	Metadata ratio:		      1.00
	Global reserve:		  95.19MiB	(used: 0.00B)
	Multiple profiles:		        no

Data,single: Size:65.44GiB, Used:49.21GiB (75.20%)
/dev/sda3 65.44GiB

Metadata,single: Size:3.00GiB, Used:2.00GiB (66.51%)
/dev/sda3 3.00GiB

System,single: Size:32.00MiB, Used:16.00KiB (0.05%)
/dev/sda3 32.00MiB

Unallocated:
/dev/sda3 1.00MiB

btrfs balance status /var/@var/
Balance on '/var/@var/' is paused
0 out of about 0 chunks balanced (0 considered), -nan% left

btrfs balance cancel /var/@var/
ERROR: balance cancel on '/var/@var/' failed: Read-only file system

Thank you for your help,

Alban

#1 lore raw

more details after a reboot (sda3 is now sdb3):

# mount -o ro,rescue=all /dev/sdb3 /var
[  597.965956] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (437)
[  597.977243] BTRFS info (device sdb3 state S): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[  597.987580] BTRFS info (device sdb3 state S): using crc32c checksum algorithm
[  598.028410] BTRFS info (device sdb3 state ECS): enabling ssd optimizations
[  598.035833] BTRFS info (device sdb3 state ECS): disabling log replay at mount time
[  598.043944] BTRFS info (device sdb3 state ECS): turning on async discard
[  598.051195] BTRFS info (device sdb3 state ECS): enabling free space tree
[  598.058445] BTRFS info (device sdb3 state ECS): ignoring bad roots
[  598.065110] BTRFS info (device sdb3 state ECS): ignoring data csums
[  598.071836] BTRFS info (device sdb3 state ECS): ignoring meta csums
[  598.078558] BTRFS info (device sdb3 state ECS): ignoring unknown super block flags
# btrfs balance resume /var
ERROR: error during balancing '/var': Read-only file system
There may be more info in syslog - try dmesg | tail
# umount /var
[ 1077.443570] BTRFS info (device sdb3 state EMCS): last unmount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
# mount -o rw,skip_balance,clear_cache /dev/sdb3 /var
[ 1084.400924] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (464)
[ 1084.412230] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[ 1084.421817] BTRFS info (device sdb3): using crc32c checksum algorithm
[ 1084.458776] BTRFS info (device sdb3): rebuilding free space tree
[ 1084.465327] BTRFS warning (device sdb3): Skipping commit of aborted transaction.
[ 1084.473209] ------------[ cut here ]------------
[ 1084.478283] BTRFS: Transaction aborted (error -28)
[ 1084.483526] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#6: kworker/u32:2/66
[ 1084.494986] Modules linked in: ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod autof
s4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper cec sd_mod rc_core drm_clie
nt_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 video realtek crc_itu_t usbcore scsi_m
od fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs                                                                                                    [ 1084.546146] CPU: 6 UID: 0 PID: 66 Comm: kworker/u32:2 Not tainted 7.0.10+deb14-amd64 #1 PREEMPT(lazy)  Debian 7.0.10-1
[ 1084.557401] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
[ 1084.564828] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[ 1084.572751] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs]
[ 1084.579439] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9 f
8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00
[ 1084.598660] RSP: 0018:ffffce9fc02fbce8 EFLAGS: 00010292
[ 1084.604372] RAX: 0000000000000002 RBX: ffff8ac4c75ae150 RCX: 0000000000000003
[ 1084.611984] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080
[ 1084.619598] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fc02fbad0
[ 1084.627200] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac4d02dd000
[ 1084.634799] R13: ffff8ac4d13a7600 R14: ffff8ac4d02dd4c0 R15: ffff8ac4c1100000
[ 1084.642401] FS:  0000000000000000(0000) GS:ffff8acc3a4fc000(0000) knlGS:0000000000000000
[ 1084.650947] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1084.657153] CR2: 00007f59bc90839c CR3: 0000000168c2c003 CR4: 00000000003706f0
[ 1084.664742] Call Trace:
[ 1084.667646]  <TASK>
[ 1084.670180]  ? start_transaction+0x228/0x880 [btrfs]
[ 1084.675649]  flush_space+0x3c3/0x5a0 [btrfs]
[ 1084.680430]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[ 1084.686591]  ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
[ 1084.693190]  do_async_reclaim_metadata_space+0x7d/0x180 [btrfs]
[ 1084.699618]  btrfs_async_reclaim_metadata_space+0x54/0x70 [btrfs]
[ 1084.706214]  process_one_work+0x19d/0x3a0
[ 1084.710679]  worker_thread+0x1af/0x320
[ 1084.714880]  ? __pfx_worker_thread+0x10/0x10
[ 1084.719592]  kthread+0xe3/0x120
[ 1084.723174]  ? __pfx_kthread+0x10/0x10
[ 1084.727370]  ret_from_fork+0x2c7/0x350
[ 1084.731562]  ? __pfx_kthread+0x10/0x10
[ 1084.735738]  ret_from_fork_asm+0x1a/0x30
[ 1084.740102]  </TASK>
[ 1084.742707] ---[ end trace 0000000000000000 ]---
[ 1084.747733] BTRFS info (device sdb3 state A): dumping space info:
[ 1084.754275] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full
[ 1084.764845] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[ 1084.779326] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full
[ 1084.789819] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[ 1084.805227] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[ 1084.815115] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[ 1084.829087] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080
[ 1084.838378] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0
[ 1084.846167] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0
[ 1084.853986] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0
[ 1084.861767] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0
[ 1084.869752] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0
[ 1084.877621] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left
[ 1084.887109] BTRFS warning (device sdb3 state EA): failed to rebuild free space tree: -28
[ 1084.895714] BTRFS error (device sdb3 state EA): commit super block returned -30
[ 1084.904462] BTRFS error (device sdb3 state EA): open_ctree failed: -28
mount: /var: fsconfig() failed: No space left on device.
       dmesg(1) may have more information after failed mount system call.

Le 09/06/2026 à 06:42, Alban Browaeys a écrit :

Quoted 167 lines

Hello,

On Debian Forky (testing), I tried to migrate my old SSD btrfs partition metadata from the previous default Single to the new DUP default.

The migration stopped halfway far /var and I am now stuck.

I am facing a severe ENOSPC deadlock it seems.

Symptoms:

- The filesystem conversion stopped midway and turned Read-Only during a balance operation.

- Attempting to remount or mount as 'rw' fails with ENOSPC.

- Adding a temporary loop device in RAM via 'btrfs device add' fails because the FS is read-only.

- Trying to run 'btrfs check --clear-space-cache v2' or cleanup tools
at cold state fails with: "error: please finish/cancel the running replace/balance before running this command".

- 'btrfs balance cancel' cannot be run because mounting 'rw' is rejected, and it refuses to target the unmounted block device directly.

I can only boot into init=/bin/sh as of now.

Here are the outputs:

btrfs fi df /var
Data, single: total=65.44GiB, used=49.21GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=3.00GiB, used=1.59GiB
GlobalReserve, single: total=181.70MiB, used=0.00B
 prahal  ~  sudo btrfs balance start -mprofiles=single -mconvert=dup /var
ERROR: error during balancing '/var': Read-only file system
There may be more info in syslog - try dmesg | tail
uin 09 02:34:34 hermes kernel: BTRFS info (device sda3): balance: start -mconvert=dup,profiles=single -sconvert=dup,profiles=single
juin 09 02:34:34 hermes kernel: BTRFS info (device sda3): relocating block group 102442729472 flags metadata
juin 09 02:35:02 hermes kernel: BTRFS error (device sda3): failed to run delayed ref for logical 3976462336 num_bytes 8192 type 184 action 1 ref_mod 1: -28
juin 09 02:35:02 hermes kernel: ------------[ cut here ]------------
juin 09 02:35:02 hermes kernel: BTRFS: Transaction aborted (error -28)
juin 09 02:35:02 hermes kernel: WARNING: fs/btrfs/extent-tree.c:2247 at btrfs_run_delayed_refs+0x115/0x120 [btrfs], CPU#0: kworker/u32:11/1370737
juin 09 02:35:02 hermes kernel: Modules linked in: nilfs2 ufs qnx4 hfsplus hfs minix msdos jfs nls_ucs2_utils xfs ext4 mbcache jbd2 crc32c_cryptoapi cpuid exfat uinput rfcomm snd_seq_dummy snd_hrtimer snd_seq x> juin 09 02:35:02 hermes kernel:  snd_soc_avs nls_ascii snd_soc_hda_codec kvm_intel snd_hda_ext_core nls_cp437 snd_hda_codec vfat snd_hda_core fat kvm btrtl btbcm snd_intel_dspcfg snd_intel_sdw_acpi snd_hwdep gs> juin 09 02:35:02 hermes kernel:  hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper sd_mod cec rc_core iTCO_wdt drm_client_lib intel_pmc_bxt xhci_pci watchdog> juin 09 02:35:02 hermes kernel: CPU: 0 UID: 0 PID: 1370737 Comm: kworker/u32:11 Tainted: P           OE       7.0.9+deb14-amd64 #1 PREEMPT(lazy)  Debian 7.0.9-1
juin 09 02:35:02 hermes kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
juin 09 02:35:02 hermes kernel: Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
juin 09 02:35:02 hermes kernel: Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
juin 09 02:35:02 hermes kernel: RIP: 0010:btrfs_run_delayed_refs+0x117/0x120 [btrfs]
juin 09 02:35:02 hermes kernel: Code: 32 27 c1 48 89 ef e8 e8 54 0f 00 eb 07 e8 f1 22 ed d5 31 db 89 d8 5b 5d 41 5c 41 5d c3 cc cc cc cc 48 8d 3d 5b 7f 31 00 89 de <67> 48 0f b9 3a eb be 66 90 90 90 90 90 90 90> juin 09 02:35:02 hermes kernel: RSP: 0018:ffffd03f75b87d48 EFLAGS: 00010292
juin 09 02:35:02 hermes kernel: RAX: 0000000000000002 RBX: 00000000ffffffe4 RCX: ffff8ad782c55600
juin 09 02:35:02 hermes kernel: RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc11dcac0
juin 09 02:35:02 hermes kernel: RBP: ffff8ad783d7bc78 R08: 0000000000000000 R09: ffffffffc0ec412a
juin 09 02:35:02 hermes kernel: R10: fffff4c487b11b40 R11: ffff8ad86c46d460 R12: 0000000000000000
juin 09 02:35:02 hermes kernel: R13: ffff8ad6a2573c00 R14: 0000000000001252 R15: 0000000000000000
juin 09 02:35:02 hermes kernel: FS:  0000000000000000(0000) GS:ffff8adee617c000(0000) knlGS:0000000000000000
juin 09 02:35:02 hermes kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
juin 09 02:35:02 hermes kernel: CR2: 00007ffd95039e68 CR3: 000000072ca2c005 CR4: 00000000003726f0
juin 09 02:35:02 hermes kernel: Call Trace:
juin 09 02:35:02 hermes kernel:  <TASK>
juin 09 02:35:02 hermes kernel:  flush_space+0x126/0x5a0 [btrfs]
juin 09 02:35:02 hermes kernel:  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
juin 09 02:35:02 hermes kernel:  ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
juin 09 02:35:02 hermes kernel: do_async_reclaim_metadata_space+0x7d/0x180 [btrfs]
juin 09 02:35:02 hermes kernel: btrfs_async_reclaim_metadata_space+0x54/0x70 [btrfs]
juin 09 02:35:02 hermes kernel:  process_one_work+0x19d/0x3a0
juin 09 02:35:02 hermes kernel:  worker_thread+0x1af/0x320
juin 09 02:35:02 hermes kernel:  ? __pfx_worker_thread+0x10/0x10
juin 09 02:35:02 hermes kernel:  kthread+0xe3/0x120
juin 09 02:35:02 hermes kernel:  ? __pfx_kthread+0x10/0x10
juin 09 02:35:02 hermes kernel:  ret_from_fork+0x2c7/0x350
juin 09 02:35:02 hermes kernel:  ? __pfx_kthread+0x10/0x10
juin 09 02:35:02 hermes kernel:  ret_from_fork_asm+0x1a/0x30
juin 09 02:35:02 hermes kernel:  </TASK>
juin 09 02:35:02 hermes kernel: ---[ end trace 0000000000000000 ]---
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): dumping space info:
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info DATA (sub-group id 0) has 17422733312 free, is full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=192512, readonly=65536 zone_unusable=0 juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info METADATA (sub-group id 0) has 442368 free, is full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=3221225472, used=2155544576, pinned=38010880, reserved=10387456, may_use=576634880, readonly=440205312 zone_unusable=0 juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info SYSTEM (sub-group id 0) has 33538048 free, is not full
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=0 zone_unusable=0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): global_block_rsv: size 190529536 reserved 190480384
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): trans_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): chunk_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): remap_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): delayed_block_rsv: size 0 reserved 0
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state A): delayed_refs_rsv: size 1270611968 reserved 368197632
juin 09 02:35:02 hermes kernel: BTRFS: error (device sda3 state A) in btrfs_run_delayed_refs:2247: errno=-28 No space left
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): forced readonly
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): 1 enospc errors during balance
juin 09 02:35:02 hermes kernel: BTRFS info (device sda3 state EA): balance: ended with status: -30

Linux (none) 7.0.10+deb14-amd64 #1 SMP PREEMPT_DYNAMIC Debian 7.0.10-1 (2026-05-27) x86_64 GNU/Linux

btrfs-progs v7.0
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices

# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type> <options>       <dump>  <pass>
LABEL=ROOT     /               btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2        0       1
UUID=9C79-15CA  /boot/efi       vfat umask=0077                    0       1
LABEL=HDDHOME     /home           btrfs defaults,compress=zstd,noatime,space_cache=v2            0       2
#LABEL=SSDHOME     /ssdhome        btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2 0       2
LABEL=SSDHOME     /var            btrfs defaults,ssd,compress=zstd,noatime,space_cache=v2,subvol=@var 0       2
UUID=EAB2-8BE4  /boot/efi-old       vfat umask=0077                    0       1

btrfs fi usage /var/@var/
Overall:
    Device size:          68.47GiB
    Device allocated:          68.47GiB
    Device unallocated:           1.00MiB
    Device missing:             0.00B
    Device slack:             0.00B
    Used:              51.21GiB
    Free (estimated):          16.23GiB    (min: 16.23GiB) Free (statfs, df):          16.23GiB
    Data ratio:                  1.00
    Metadata ratio:              1.00
    Global reserve:          95.19MiB    (used: 0.00B) Multiple profiles:                no

Data,single: Size:65.44GiB, Used:49.21GiB (75.20%) /dev/sda3      65.44GiB

Metadata,single: Size:3.00GiB, Used:2.00GiB (66.51%) /dev/sda3       3.00GiB

System,single: Size:32.00MiB, Used:16.00KiB (0.05%) /dev/sda3      32.00MiB

Unallocated:
   /dev/sda3       1.00MiB

btrfs balance status /var/@var/
Balance on '/var/@var/' is paused
0 out of about 0 chunks balanced (0 considered), -nan% left

btrfs balance cancel /var/@var/
ERROR: balance cancel on '/var/@var/' failed: Read-only file system

Thank you for your help,

Alban

#2 lore raw

Try:

mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var btrfs balance start -dlimit=1 /var

If the first command works but the second command does not, you could insert a btrfs device add command to add a partition. I'm not such a big fan of adding USB sticks or zram devices because if anything goes wrong, the file system ends up broken.

This is a bit of a heavy hammer to fix this problem but it might actually be faster than troubleshooting Btrfs enospc:

It should still be possible to make the troubled Btrfs a read-only seed device using btrfs-tune; then mount it ro; use btrfs dev add using a device that's equal to or bigger; remount rw; then btrfs dev rem the 1st device, i.e. the ro seed. This will replicate/clone the file system from device 1 to device 2. And it'll take a while. But in the end you'll have a balanced file system on device 2.

It isn't strictly necessary for device 2 to be equal to or bigger. It is possible to add a smaller device, but before you remove the 1st device, you wlil need to delete files. The bigger files you choose to delete, the fewer you will need to delete to get to the underprovisioned 2nd device size. Then remove the 1st device. Now only the files that have not been deleted will have extents replicated to the 2nd device.

At the very least the seed-device is not being modified and shouldn't get any worse than it is.

Chris Murphy

#3 lore raw

Le 09/06/2026 à 17:25, Chris Murphy a écrit :

Quoted 4 lines

Try:

mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var btrfs balance start -dlimit=1 /var

Thanks for the suggestions. Unfortunately, the bypass did not work.

# mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var
[117052.507168] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (2004)
[117052.526475] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[117052.536146] BTRFS info (device sdb3): using crc32c checksum algorithm
[117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree
[117052.550097] BTRFS error (device sdb3): open_ctree failed: -22
mount: /var: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

I believe this issue is blocking me:

[117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree

So I tried with enospc_debug without nospace_cache, to get debug infos:

# mount -o skip_balance,enospc_debug /dev/sdb3 /var
[117261.590047] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (2008)
[117261.612863] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[117261.622548] BTRFS info (device sdb3): using crc32c checksum algorithm
[117261.662415] BTRFS info (device sdb3): left=0, need=393216, flags=36
[117261.669328] BTRFS info (device sdb3): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[117261.678792] BTRFS info (device sdb3): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[117261.692344] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190382080
[117261.701131] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0
[117261.708349] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0
[117261.715571] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0
[117261.722782] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0
[117261.730154] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0
[117261.737462] BTRFS error (device sdb3): allocation failed flags 36, wanted 16384 tree-log 0, relocation: 0
[117261.747497] BTRFS info (device sdb3): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117261.757371] BTRFS info (device sdb3): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117261.772172] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190365696
[117261.780844] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0
[117261.787995] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0
[117261.795127] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0
[117261.802251] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0
[117261.809551] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0
[117261.816785] BTRFS info (device sdb3): block group 55856594944 has 1073741824 bytes, 1007976448 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (65765376
 bytes available) [readonly]
[117261.834327] BTRFS info (device sdb3): block group has cluster?: no
[117261.840947] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes
[117261.849500] BTRFS info (device sdb3): block group 101368987648 has 1073741824 bytes, 500957184 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (57278464
0 bytes available) [readonly]
[117261.867163] BTRFS info (device sdb3): block group has cluster?: no
[117261.873784] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes                                                                            [117261.882313] BTRFS info (device sdb3): block group 102442729472 has 1073741824 bytes, 633536512 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (44020531
2 bytes available) [readonly]
[117261.899994] BTRFS info (device sdb3): block group has cluster?: no
[117261.906630] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes
[117261.915176] BTRFS info (device sdb3): 1078755328 bytes available across all block groups
[117261.923750] BTRFS warning (device sdb3): Skipping commit of aborted transaction.
[117261.931608] ------------[ cut here ]------------
[117261.936687] BTRFS: Transaction aborted (error -28)
[117261.941950] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#6: mount/2008
[117261.952905] Modules linked in: ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod auto
fs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper cec sd_mod rc_core drm_cli
ent_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 video realtek crc_itu_t usbcore scsi_
mod fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs
[117262.004021] CPU: 6 UID: 0 PID: 2008 Comm: mount Tainted: G        W           7.0.10+deb14-amd64 #1 PREEMPT(lazy)  Debian 7.0.10-1
[117262.016343] Tainted: [W]=WARN
[117262.019853] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
[117262.027348] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs]
[117262.034117] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9
f8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00
[117262.053426] RSP: 0018:ffffce9fc8b57418 EFLAGS: 00010292
[117262.059233] RAX: 0000000000000002 RBX: ffff8ac4d181a738 RCX: 0000000000000003
[117262.066946] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080
[117262.074658] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fc8b57200
[117262.082364] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac545885000
[117262.090075] R13: ffff8ac4c242ea00 R14: ffff8ac5458854c0 R15: ffff8ac4cff01b80
[117262.097779] FS:  00007f22c28cf940(0000) GS:ffff8acc3a4fc000(0000) knlGS:0000000000000000
[117262.106429] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[117262.112740] CR2: 00007f22c2cdfdd0 CR3: 00000001867ac001 CR4: 00000000003706f0
[117262.120434] Call Trace:
[117262.123451]  <TASK>
[117262.126121]  ? start_transaction+0x228/0x880 [btrfs]
[117262.131676]  flush_space+0x3c3/0x5a0 [btrfs]
[117262.136574]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[117262.142855]  ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
[117262.149597]  priority_reclaim_metadata_space+0x96/0x130 [btrfs]
[117262.156132]  handle_reserve_ticket+0x5b/0x290 [btrfs]
[117262.161800]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[117262.168076]  reserve_bytes+0x268/0x4c0 [btrfs]                                                                                                                      [117262.173127]  btrfs_reserve_metadata_bytes+0x1a/0xe0 [btrfs]
[117262.179306]  btrfs_block_rsv_refill+0x60/0x90 [btrfs]
[117262.184965]  evict_refill_and_join+0x47/0xe0 [btrfs]
[117262.190531]  btrfs_evict_inode+0x332/0x400 [btrfs]
[117262.195922]  evict+0xfd/0x270
[117262.199441]  btrfs_orphan_cleanup+0x21f/0x390 [btrfs]
[117262.205085]  btrfs_start_pre_rw_mount+0x2a0/0x6c0 [btrfs]
[117262.211071]  open_ctree+0x12fb/0x1730 [btrfs]
[117262.216028]  btrfs_get_tree.cold+0xb/0xb6 [btrfs]
[117262.221324]  ? security_inode_permission+0x4a/0x110
[117262.226747]  vfs_get_tree+0x29/0xd0
[117262.230776]  vfs_cmd_create+0x57/0xd0
[117262.234947]  __do_sys_fsconfig+0x2fc/0x5e0
[117262.239581]  do_syscall_64+0xcd/0x15f0
[117262.243873]  ? __fs_parse+0x68/0x1c0
[117262.247989]  ? aa_file_perm+0x18a/0x550
[117262.252368]  ? btrfs_parse_param+0x4f/0x960 [btrfs]
[117262.257818]  ? apparmor_file_permission+0x44/0xb0
[117262.263052]  ? fscontext_read+0x171/0x190
[117262.267594]  ? rw_verify_area+0x56/0x180
[117262.272039]  ? vfs_read+0xbf/0x390
[117262.275964]  ? __fs_parse+0x68/0x1c0
[117262.280063]  ? aa_file_perm+0x18a/0x550
[117262.284422]  ? btrfs_parse_param+0x4f/0x960 [btrfs]
[117262.289858]  ? apparmor_file_permission+0x44/0xb0
[117262.295082]  ? fscontext_read+0x171/0x190
[117262.299605]  ? rw_verify_area+0x56/0x180
[117262.304036]  ? vfs_read+0xbf/0x390
[117262.307943]  ? do_syscall_64+0x10a/0x15f0
[117262.312451]  ? from_kgid_munged+0x17/0x30
[117262.316955]  ? __do_sys_getgid+0x27/0x30
[117262.321369]  ? do_syscall_64+0x10a/0x15f0
[117262.325856]  ? from_kgid_munged+0x17/0x30
[117262.330340]  ? ksys_read+0x73/0xf0
[117262.334213]  ? do_syscall_64+0x10a/0x15f0
[117262.338694]  ? irqentry_exit+0x78/0x680
[117262.343000]  ? exc_page_fault+0x82/0x1d0
[117262.347384]  ? __irq_exit_rcu+0x3d/0xe0                                                                                                                             [117262.351683]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[117262.357197] RIP: 0033:0x7f22c2d0eb4a
[117262.361218] Code: 73 01 c3 48 8b 0d c6 32 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff
73 01 c3 48 8b 0d 96 32 0d 00 f7 d8 64 89 01 48
[117262.380431] RSP: 002b:00007ffd5d547f68 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
[117262.388464] RAX: ffffffffffffffda RBX: 0000557471e4da50 RCX: 00007f22c2d0eb4a
[117262.396057] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
[117262.403650] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
[117262.411240] R10: 0000000000000000 R11: 0000000000000246 R12: 0000557471e4ec40
[117262.418831] R13: 00007f22c2ea127c R14: 00007f22c2e9f460 R15: 00007f22c28cf7f0
[117262.426410]  </TASK>
[117262.429024] ---[ end trace 0000000000000000 ]---
[117262.434074] BTRFS info (device sdb3 state A): dumping space info:
[117262.440595] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full
[117262.451184] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[117262.465659] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.476171] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.491598] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[117262.501477] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[117262.515439] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080
[117262.524731] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0
[117262.532538] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0
[117262.540337] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0
[117262.548136] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0
[117262.556110] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0
[117262.563998] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left
[117262.573453] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.584033] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.599563] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080
[117262.608943] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0
[117262.616846] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0
[117262.624749] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0
[117262.632643] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0
[117262.640715] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0
[117262.648696] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.659312] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.674875] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080
[117262.684287] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0
[117262.692225] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0
[117262.700155] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0
[117262.708086] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0
[117262.716191] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0
[117262.724260] BTRFS warning (device sdb3 state EA): could not allocate space for delete; will truncate on mount
[117262.734622] BTRFS error (device sdb3 state EA): Error removing orphan entry, stopping orphan cleanup
[117262.744220] BTRFS error (device sdb3 state EA): could not do orphan cleanup -28
[117262.752021] BTRFS error (device sdb3 state EA): commit super block returned -30
[117262.760739] BTRFS error (device sdb3 state EA): open_ctree failed: -28
mount: /var: fsconfig() failed: No space left on device.
       dmesg(1) may have more information after failed mount system call.
Quoted 3 lines

If the first command works but the second command does not, you could insert a btrfs device add command to add a partition. I'm not such a big fan of adding USB sticks or zram devices because if anything goes wrong, the file system ends up broken.

This is a bit of a heavy hammer to fix this problem but it might actually be faster than troubleshooting Btrfs enospc:

I am all for troubleshooting this enospc issue because it is the first time I cannot even add a device to fix the enospc... in fact an hour before this issue I had an enospc issue this time with the root partition that had very few space left, and adding a device worked well and it was pretty easy to fix. But here I have gigabytes of unused data and metadata, I am puzzled.

I was switching my SSD btrfs partitions (I plan to merge them all but for historical reason they are split as separate btrfs partitions instead of subvolumes of a single btrfs partition) from single to DUP, the new default.

 I believe the main issue here is that the conversion attempted to use until the last megabyte of metadata before errorring out, preventing any simple rescue operation afterwards, and even preventing from balancing data to free chunks for metadata.

It would have been fine for the conversion to error out and let me balance data then resume the conversion. The main issue is that the conversion used so much metadata space that I cannot even mount the btrfs partition to balance its data space.

Maybe the metadata conversion should guard/ straight error out when there is not enough space in metadata (here 3GB to duplicate 1,59GB does not fit) and there is no way to get a new free chunk for metadata and tell to free chunks first ?

Quoted 6 lines

It should still be possible to make the troubled Btrfs a read-only seed device using btrfs-tune; then mount it ro; use btrfs dev add using a device that's equal to or bigger; remount rw; then btrfs dev rem the 1st device, i.e. the ro seed. This will replicate/clone the file system from device 1 to device 2. And it'll take a while. But in the end you'll have a balanced file system on device 2.

It isn't strictly necessary for device 2 to be equal to or bigger. It is possible to add a smaller device, but before you remove the 1st device, you wlil need to delete files. The bigger files you choose to delete, the fewer you will need to delete to get to the underprovisioned 2nd device size. Then remove the 1st device. Now only the files that have not been deleted will have extents replicated to the 2nd device.

At the very least the seed-device is not being modified and shouldn't get any worse than it is. Chris Murphy

I don't know yet how to proceed with such a read-only seed option, I will look into it. But is there other flags or tune I could do that could let me cancel the pause operation, the balance the data space, then retry the conversion ?

Alban

#4 lore raw

Here is the debug dump of this partition sdb3_var_metadata-only-debug.img at https://drive.google.com/file/d/1OzEZLaTUNpWsL1ohGKOMW4GiR9oQakdP/view?usp=drive_link done with "btrfs-image -c 9 -t 4 /dev/sdb3 /mnt/4/hermes-var-20260610-enospc-vanilla/sdb3_var_metadata-only-debug.img".

I tried to btrfs-resotre this partition but I get a lot of errors:

btrfs restore -v -m -x -i /dev/sdb3 /mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/

Skipping existing file /mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/@var/backups/group.bak If you wish to overwrite use -o
ERROR: zstd frame incomplete
ERROR: copying data for /mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/@var/backups/dpkg.arch.5.gz failed
ERROR: zstd frame incomplete
(...)

Alban

Le 10/06/2026 à 23:05, Alban Browaeys a écrit :

Quoted 212 lines

Le 09/06/2026 à 17:25, Chris Murphy a écrit :

Quoted 4 lines

Try:

mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var btrfs balance start -dlimit=1 /var

Thanks for the suggestions. Unfortunately, the bypass did not work.

# mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var
[117052.507168] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (2004)
[117052.526475] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[117052.536146] BTRFS info (device sdb3): using crc32c checksum algorithm
[117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree
[117052.550097] BTRFS error (device sdb3): open_ctree failed: -22
mount: /var: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

I believe this issue is blocking me:

[117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree

So I tried with enospc_debug without nospace_cache, to get debug infos:

# mount -o skip_balance,enospc_debug /dev/sdb3 /var
[117261.590047] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (2008)
[117261.612863] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[117261.622548] BTRFS info (device sdb3): using crc32c checksum algorithm
[117261.662415] BTRFS info (device sdb3): left=0, need=393216, flags=36
[117261.669328] BTRFS info (device sdb3): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[117261.678792] BTRFS info (device sdb3): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[117261.692344] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190382080
[117261.701131] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0
[117261.708349] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0
[117261.715571] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0
[117261.722782] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0
[117261.730154] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0
[117261.737462] BTRFS error (device sdb3): allocation failed flags 36, wanted 16384 tree-log 0, relocation: 0
[117261.747497] BTRFS info (device sdb3): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117261.757371] BTRFS info (device sdb3): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117261.772172] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190365696
[117261.780844] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0
[117261.787995] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0
[117261.795127] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0
[117261.802251] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0
[117261.809551] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0
[117261.816785] BTRFS info (device sdb3): block group 55856594944 has 1073741824 bytes, 1007976448 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (65765376
 bytes available) [readonly]
[117261.834327] BTRFS info (device sdb3): block group has cluster?: no
[117261.840947] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes
[117261.849500] BTRFS info (device sdb3): block group 101368987648 has 1073741824 bytes, 500957184 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (57278464
0 bytes available) [readonly]
[117261.867163] BTRFS info (device sdb3): block group has cluster?: no
[117261.873784] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.882313] BTRFS info (device sdb3): block group 102442729472 has 1073741824 bytes, 633536512 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (44020531
2 bytes available) [readonly]
[117261.899994] BTRFS info (device sdb3): block group has cluster?: no
[117261.906630] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes
[117261.915176] BTRFS info (device sdb3): 1078755328 bytes available across all block groups
[117261.923750] BTRFS warning (device sdb3): Skipping commit of aborted transaction.
[117261.931608] ------------[ cut here ]------------
[117261.936687] BTRFS: Transaction aborted (error -28)
[117261.941950] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#6: mount/2008
[117261.952905] Modules linked in: ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod auto
fs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper cec sd_mod rc_core drm_cli
ent_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 video realtek crc_itu_t usbcore scsi_
mod fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs
[117262.004021] CPU: 6 UID: 0 PID: 2008 Comm: mount Tainted: G        W           7.0.10+deb14-amd64 #1 PREEMPT(lazy)  Debian 7.0.10-1
[117262.016343] Tainted: [W]=WARN
[117262.019853] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
[117262.027348] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs]
[117262.034117] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9
f8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00
[117262.053426] RSP: 0018:ffffce9fc8b57418 EFLAGS: 00010292
[117262.059233] RAX: 0000000000000002 RBX: ffff8ac4d181a738 RCX: 0000000000000003
[117262.066946] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080
[117262.074658] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fc8b57200
[117262.082364] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac545885000
[117262.090075] R13: ffff8ac4c242ea00 R14: ffff8ac5458854c0 R15: ffff8ac4cff01b80
[117262.097779] FS:  00007f22c28cf940(0000) GS:ffff8acc3a4fc000(0000) knlGS:0000000000000000
[117262.106429] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[117262.112740] CR2: 00007f22c2cdfdd0 CR3: 00000001867ac001 CR4: 00000000003706f0
[117262.120434] Call Trace:
[117262.123451]  <TASK>
[117262.126121]  ? start_transaction+0x228/0x880 [btrfs]
[117262.131676]  flush_space+0x3c3/0x5a0 [btrfs]
[117262.136574]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[117262.142855]  ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
[117262.149597]  priority_reclaim_metadata_space+0x96/0x130 [btrfs]
[117262.156132]  handle_reserve_ticket+0x5b/0x290 [btrfs]
[117262.161800]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[117262.168076]  reserve_bytes+0x268/0x4c0 [btrfs] [117262.173127]  btrfs_reserve_metadata_bytes+0x1a/0xe0 [btrfs]
[117262.179306]  btrfs_block_rsv_refill+0x60/0x90 [btrfs]
[117262.184965]  evict_refill_and_join+0x47/0xe0 [btrfs]
[117262.190531]  btrfs_evict_inode+0x332/0x400 [btrfs]
[117262.195922]  evict+0xfd/0x270
[117262.199441]  btrfs_orphan_cleanup+0x21f/0x390 [btrfs]
[117262.205085]  btrfs_start_pre_rw_mount+0x2a0/0x6c0 [btrfs]
[117262.211071]  open_ctree+0x12fb/0x1730 [btrfs]
[117262.216028]  btrfs_get_tree.cold+0xb/0xb6 [btrfs]
[117262.221324]  ? security_inode_permission+0x4a/0x110
[117262.226747]  vfs_get_tree+0x29/0xd0
[117262.230776]  vfs_cmd_create+0x57/0xd0
[117262.234947]  __do_sys_fsconfig+0x2fc/0x5e0
[117262.239581]  do_syscall_64+0xcd/0x15f0
[117262.243873]  ? __fs_parse+0x68/0x1c0
[117262.247989]  ? aa_file_perm+0x18a/0x550
[117262.252368]  ? btrfs_parse_param+0x4f/0x960 [btrfs]
[117262.257818]  ? apparmor_file_permission+0x44/0xb0
[117262.263052]  ? fscontext_read+0x171/0x190
[117262.267594]  ? rw_verify_area+0x56/0x180
[117262.272039]  ? vfs_read+0xbf/0x390
[117262.275964]  ? __fs_parse+0x68/0x1c0
[117262.280063]  ? aa_file_perm+0x18a/0x550
[117262.284422]  ? btrfs_parse_param+0x4f/0x960 [btrfs]
[117262.289858]  ? apparmor_file_permission+0x44/0xb0
[117262.295082]  ? fscontext_read+0x171/0x190
[117262.299605]  ? rw_verify_area+0x56/0x180
[117262.304036]  ? vfs_read+0xbf/0x390
[117262.307943]  ? do_syscall_64+0x10a/0x15f0
[117262.312451]  ? from_kgid_munged+0x17/0x30
[117262.316955]  ? __do_sys_getgid+0x27/0x30
[117262.321369]  ? do_syscall_64+0x10a/0x15f0
[117262.325856]  ? from_kgid_munged+0x17/0x30
[117262.330340]  ? ksys_read+0x73/0xf0
[117262.334213]  ? do_syscall_64+0x10a/0x15f0
[117262.338694]  ? irqentry_exit+0x78/0x680
[117262.343000]  ? exc_page_fault+0x82/0x1d0
[117262.347384]  ? __irq_exit_rcu+0x3d/0xe0 [117262.351683]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[117262.357197] RIP: 0033:0x7f22c2d0eb4a
[117262.361218] Code: 73 01 c3 48 8b 0d c6 32 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff
73 01 c3 48 8b 0d 96 32 0d 00 f7 d8 64 89 01 48
[117262.380431] RSP: 002b:00007ffd5d547f68 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
[117262.388464] RAX: ffffffffffffffda RBX: 0000557471e4da50 RCX: 00007f22c2d0eb4a
[117262.396057] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
[117262.403650] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
[117262.411240] R10: 0000000000000000 R11: 0000000000000246 R12: 0000557471e4ec40
[117262.418831] R13: 00007f22c2ea127c R14: 00007f22c2e9f460 R15: 00007f22c28cf7f0
[117262.426410]  </TASK>
[117262.429024] ---[ end trace 0000000000000000 ]---
[117262.434074] BTRFS info (device sdb3 state A): dumping space info:
[117262.440595] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full
[117262.451184] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[117262.465659] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.476171] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.491598] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[117262.501477] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[117262.515439] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080
[117262.524731] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0
[117262.532538] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0
[117262.540337] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0
[117262.548136] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0
[117262.556110] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0
[117262.563998] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left
[117262.573453] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.584033] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.599563] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080
[117262.608943] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0
[117262.616846] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0
[117262.624749] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0
[117262.632643] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0
[117262.640715] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0
[117262.648696] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.659312] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.674875] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080
[117262.684287] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0
[117262.692225] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0
[117262.700155] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0
[117262.708086] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0
[117262.716191] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0
[117262.724260] BTRFS warning (device sdb3 state EA): could not allocate space for delete; will truncate on mount
[117262.734622] BTRFS error (device sdb3 state EA): Error removing orphan entry, stopping orphan cleanup
[117262.744220] BTRFS error (device sdb3 state EA): could not do orphan cleanup -28
[117262.752021] BTRFS error (device sdb3 state EA): commit super block returned -30
[117262.760739] BTRFS error (device sdb3 state EA): open_ctree failed: 
-28
mount: /var: fsconfig() failed: No space left on device.
       dmesg(1) may have more information after failed mount system call.
Quoted 3 lines

If the first command works but the second command does not, you could insert a btrfs device add command to add a partition. I'm not such a big fan of adding USB sticks or zram devices because if anything goes wrong, the file system ends up broken.

This is a bit of a heavy hammer to fix this problem but it might actually be faster than troubleshooting Btrfs enospc:

I am all for troubleshooting this enospc issue because it is the first time I cannot even add a device to fix the enospc... in fact an hour before this issue I had an enospc issue this time with the root partition that had very few space left, and adding a device worked well and it was pretty easy to fix. But here I have gigabytes of unused data and metadata, I am puzzled.

I was switching my SSD btrfs partitions (I plan to merge them all but for historical reason they are split as separate btrfs partitions instead of subvolumes of a single btrfs partition) from single to DUP, the new default.

 I believe the main issue here is that the conversion attempted to use until the last megabyte of metadata before errorring out, preventing any simple rescue operation afterwards, and even preventing from balancing data to free chunks for metadata.

It would have been fine for the conversion to error out and let me balance data then resume the conversion. The main issue is that the conversion used so much metadata space that I cannot even mount the btrfs partition to balance its data space.

Maybe the metadata conversion should guard/ straight error out when there is not enough space in metadata (here 3GB to duplicate 1,59GB does not fit) and there is no way to get a new free chunk for metadata and tell to free chunks first ?

Quoted 6 lines

It should still be possible to make the troubled Btrfs a read-only seed device using btrfs-tune; then mount it ro; use btrfs dev add using a device that's equal to or bigger; remount rw; then btrfs dev rem the 1st device, i.e. the ro seed. This will replicate/clone the file system from device 1 to device 2. And it'll take a while. But in the end you'll have a balanced file system on device 2.

It isn't strictly necessary for device 2 to be equal to or bigger. It is possible to add a smaller device, but before you remove the 1st device, you wlil need to delete files. The bigger files you choose to delete, the fewer you will need to delete to get to the underprovisioned 2nd device size. Then remove the 1st device. Now only the files that have not been deleted will have extents replicated to the 2nd device.

At the very least the seed-device is not being modified and shouldn't get any worse than it is. Chris Murphy

I don't know yet how to proceed with such a read-only seed option, I will look into it. But is there other flags or tune I could do that could let me cancel the pause operation, the balance the data space, then retry the conversion ?

Alban

#5 lore raw

Hi,

I had space after the sdb3 btrfs /var partition, a 9GB swap partition. I deleted this swap partition and expanded the sdb3 btrfs size with parted But I don't seem to be able to expand the btrfs filesystem inside sdb3 if I cannot mount it read write, is it so? Or is it only due to my pending operation? Can I get this btrfs partition back?

I have an image of this partiton but it seems I will have the same issue with this image and will be unable to restore it.
Is there a way to copy the content of the partition mounted in ro in its current state, create a new partition and copy the content back to it?
There are docker subvolumes on it wich seems they can be recreated by docker later on (I hope so) so rsync might be an option but if the docker subvolume need to be backed up rsync is not able to backup them correctly. And btrfs restore gives me thousands of

Also should I expect this behavior to happen everytime I do a btrfs filessytem conversion without checking I have enough metadata space available beforehand or will this be prevented by some new code one day ?

# parted /dev/sdb
GNU Parted 3.7
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) select /dev/sdb
Using /dev/sdb
(parted) p
Model: ATA SanDisk SD9SB8W1 (scsi)
Disk /dev/sdb: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number  Start   End     Size    File system     Name                  Flags
 1      1049kB  903MB   902MB   fat32           EFI system partition 
boot, hidden, esp
 2      903MB   39.8GB  38.9GB  btrfs           root
 3      43.2GB  117GB   73.5GB  btrfs           ssdhome
 4      119GB   128GB   9197MB  linux-swap(v1)  SSDSWAP               swap

(parted) rm 4
(parted) resizepart 3 100%
(parted) p
Model: ATA SanDisk SD9SB8W1 (scsi)
Disk /dev/sdb: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name                  Flags
 1      1049kB  903MB   902MB   fat32        EFI system partition 
boot, hidden, esp
 2      903MB   39.8GB  38.9GB  btrfs        root
 3      43.2GB  128GB   84.8GB  btrfs        ssdhome
# mount /dev/sdb3 /mnt/3 
[441290.450174] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (7976) 
                          [441290.461454] BTRFS info (device sdb3): 
first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[441290.471065] BTRFS info (device sdb3): using crc32c checksum algorithm 
[441290.518919] BTRFS warning (device sdb3): Skipping commit of aborted transaction. 
[441290.527296] ------------[ cut here ]------------ 
[441290.532875] BTRFS: Transaction aborted (error -28) 
[441290.538625] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#2: mount/7976
[441290.550177] Modules linked in: nls_ascii nls_cp437 vfat fat exfat ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx x

or raid6_pq raid1 raid0 md_mod autofs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_displa y_helper cec sd_mod rc_core drm_client_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 vi deo realtek crc_itu_t usbcore scsi_mod fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs

[441290.605209] CPU: 2 UID: 0 PID: 7976 Comm: mount Tainted: G        W 
         7.0.10+deb14-amd64 #1 PREEMPT(lazy)  Debian 7.0.10-1 
[441290.617556] Tainted: [W]=WARN 
[441290.621060] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
[441290.628556] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs] 
[441290.635306] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9
f8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00
[441290.654597] RSP: 0018:ffffce9fcc3fb6b8 EFLAGS: 00010292
[441290.660386] RAX: 0000000000000002 RBX: ffff8ac4c06a1c78 RCX: 0000000000000003
[441290.668074] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080
[441290.675771] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fcc3fb4a0
[441290.683464] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac4c584f000
[441290.691152] R13: ffff8ac4c7211600 R14: ffff8ac4c584f4c0 R15: ffff8ac545b71b80
[441290.698842] FS:  00007f16a636e940(0000) GS:ffff8acc3a3fc000(0000) knlGS:0000000000000000 
[441290.707489] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
                       [441290.713789] CR2: 00007f71f5dcb19c CR3: 
000000034a2fa002 CR4: 00000000003706f0
[441290.721478] Call Trace: 
[441290.724485]  <TASK> 
[441290.727145]  ? start_transaction+0x228/0x880 [btrfs] 
[441290.732712]  flush_space+0x3c3/0x5a0 [btrfs] 
[441290.737591]  ? __schedule+0x489/0x1010 
[441290.741896]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs] 
[441290.748152]  ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs] 
[441290.754844]  priority_reclaim_metadata_space+0x96/0x130 [btrfs]
[441290.761360]  handle_reserve_ticket+0x5b/0x290 [btrfs]
[441290.767011]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs] 
[441290.773268]  reserve_bytes+0x268/0x4c0 [btrfs] 
[441290.778314]  btrfs_reserve_metadata_bytes+0x1a/0xe0 [btrfs] 
[441290.784484]  btrfs_block_rsv_refill+0x60/0x90 [btrfs] 
                       [441290.790127]  evict_refill_and_join+0x47/0xe0 

[btrfs]

[441290.795671]  btrfs_evict_inode+0x332/0x400 [btrfs] 
[441290.801048]  evict+0xfd/0x270 
[441290.804556]  btrfs_orphan_cleanup+0x21f/0x390 [btrfs] 
[441290.810191]  btrfs_start_pre_rw_mount+0x2a0/0x6c0 [btrfs] 
[441290.816159]  open_ctree+0x12fb/0x1730 [btrfs] 
[441290.821099]  btrfs_get_tree.cold+0xb/0xb6 [btrfs] 
[441290.826385]  ? ksys_read+0x73/0xf0 
[441290.830311]  vfs_get_tree+0x29/0xd0 
[441290.834324]  vfs_cmd_create+0x57/0xd0 
[441290.838474]  __do_sys_fsconfig+0x2fc/0x5e0 
[441290.843059]  do_syscall_64+0xcd/0x15f0 
[441290.847326]  ? from_kuid_munged+0x13/0x30 
[441290.851847]  ? __do_sys_geteuid+0x27/0x30 
[441290.856373]  ? do_syscall_64+0x10a/0x15f0 
[441290.860889]  ? __x64_sys_mkdir+0x39/0x50
[441290.865317]  ? do_syscall_64+0x10a/0x15f0
[441290.869841]  ? __x64_sys_statx+0x73/0xd0
[441290.874261]  ? do_syscall_64+0x10a/0x15f0 
[441290.878766]  ? irqentry_exit+0x78/0x680 
[441290.883089]  ? exc_page_fault+0x82/0x1d0 
[441290.887504]  entry_SYSCALL_64_after_hwframe+0x76/0x7e 
[441290.893048] RIP: 0033:0x7f16a67adb4a 
[441290.897115] Code: 73 01 c3 48 8b 0d c6 32 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 96 32 0d 00 f7 d8 64 89 01 48 
[441290.916373] RSP: 002b:00007ffded6881b8 EFLAGS: 00000246 ORIG_RAX: 00000000000001af 
[441290.924455] RAX: ffffffffffffffda RBX: 00005603bb8eba50 RCX: 00007f16a67adb4a
[441290.932099] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
[441290.939743] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
[441290.947387] R10: 0000000000000000 R11: 0000000000000246 R12: 00005603bb8ecb80
[441290.955041] R13: 00007f16a694027c R14: 00007f16a693e460 R15: 00007f16a636e7f0
[441290.962695]  </TASK> 
[441290.965396] ---[ end trace 0000000000000000 ]---
[441290.970507] BTRFS info (device sdb3 state A): dumping space info:
[441290.977124] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full 
[441290.987784] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[441291.002378] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full 
[441291.012959] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0 
[441291.028472] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full 
[441291.038428] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[441291.052461] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080 
[441291.061805] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0
[441291.069675] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0 
[441291.077552] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0 
[441291.085436] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0 
[441291.093455] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0 
[441291.101367] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left 
[441291.110841] BTRFS warning (device sdb3 state EA): could not allocate space for delete; will truncate on mount 
[441291.121191] BTRFS error (device sdb3 state EA): Error removing orphan entry, stopping orphan cleanup 
[441291.130780] BTRFS error (device sdb3 state EA): could not do orphan cleanup -28
[441291.138588] BTRFS error (device sdb3 state EA): commit super block returned -30
[441291.147315] BTRFS error (device sdb3 state EA): open_ctree failed: -28 

mount: /mnt/3: fsconfig() failed: No space left on device.
dmesg(1) may have more information after failed mount system call.

# mount -o ro,rescue=all /dev/sdb3 /var
[442011.834443] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (8020)
[442011.845803] BTRFS info (device sdb3 state S): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[442011.856089] BTRFS info (device sdb3 state S): using crc32c checksum algorithm
[442011.897204] BTRFS info (device sdb3 state ECS): enabling ssd optimizations
[442011.904566] BTRFS info (device sdb3 state ECS): disabling log replay at mount time
[442011.912626] BTRFS info (device sdb3 state ECS): turning on async discard
[442011.919811] BTRFS info (device sdb3 state ECS): enabling free space tree
[442011.927008] BTRFS info (device sdb3 state ECS): ignoring bad roots
[442011.933661] BTRFS info (device sdb3 state ECS): ignoring data csums
[442011.940357] BTRFS info (device sdb3 state ECS): ignoring meta csums
[442011.947051] BTRFS info (device sdb3 state ECS): ignoring unknown super block flags
# ls /var
@var
# btrfs fi resize max /var
ERROR: unable to start resize, another exclusive operation 'balance paused' in progress

Le 11/06/2026 à 23:19, Alban Browaeys a écrit :

Quoted 13 lines

Here is the debug dump of this partition sdb3_var_metadata-only- debug.img at https://drive.google.com/file/ d/1OzEZLaTUNpWsL1ohGKOMW4GiR9oQakdP/view?usp=drive_link done with "btrfs-image -c 9 -t 4 /dev/sdb3 /mnt/4/hermes-var-20260610-enospc- vanilla/sdb3_var_metadata-only-debug.img".

I tried to btrfs-resotre this partition but I get a lot of errors:

btrfs restore -v -m -x -i /dev/sdb3 /mnt/4/hermes-var-20260610-enospc- vanilla/restored_var/

Skipping existing file /mnt/4/hermes-var-20260610-enospc-vanilla/ restored_var/@var/backups/group.bak If you wish to overwrite use -o
ERROR: zstd frame incomplete
ERROR: copying data for /mnt/4/hermes-var-20260610-enospc-vanilla/ restored_var/@var/backups/dpkg.arch.5.gz failed
ERROR: zstd frame incomplete
(...)

Quoted 208 lines

Alban

Le 10/06/2026 à 23:05, Alban Browaeys a écrit :

Quoted 205 lines

Le 09/06/2026 à 17:25, Chris Murphy a écrit :

Quoted 4 lines

Try:

mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var btrfs balance start -dlimit=1 /var

Thanks for the suggestions. Unfortunately, the bypass did not work.

# mount -o skip_balance,nospace_cache,enospc_debug /dev/sdb3 /var
[117052.507168] BTRFS: device label SSDHOME devid 1 transid 44655875 / dev/sdb3 (8:19) scanned by mount (2004)
[117052.526475] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[117052.536146] BTRFS info (device sdb3): using crc32c checksum algorithm
[117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree
[117052.550097] BTRFS error (device sdb3): open_ctree failed: -22
mount: /var: wrong fs type, bad option, bad superblock on /dev/sdb3, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

I believe this issue is blocking me:

[117052.543125] BTRFS error (device sdb3): cannot disable free-space-tree

So I tried with enospc_debug without nospace_cache, to get debug infos:

# mount -o skip_balance,enospc_debug /dev/sdb3 /var
[117261.590047] BTRFS: device label SSDHOME devid 1 transid 44655875 / dev/sdb3 (8:19) scanned by mount (2008)
[117261.612863] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[117261.622548] BTRFS info (device sdb3): using crc32c checksum algorithm
[117261.662415] BTRFS info (device sdb3): left=0, need=393216, flags=36
[117261.669328] BTRFS info (device sdb3): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[117261.678792] BTRFS info (device sdb3): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[117261.692344] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190382080
[117261.701131] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0
[117261.708349] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0
[117261.715571] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0
[117261.722782] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0
[117261.730154] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0
[117261.737462] BTRFS error (device sdb3): allocation failed flags 36, wanted 16384 tree-log 0, relocation: 0
[117261.747497] BTRFS info (device sdb3): space_info METADATA (sub- group id 0) has -190382080 free, is full
[117261.757371] BTRFS info (device sdb3): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117261.772172] BTRFS info (device sdb3): global_block_rsv: size 190382080 reserved 190365696
[117261.780844] BTRFS info (device sdb3): trans_block_rsv: size 0 reserved 0
[117261.787995] BTRFS info (device sdb3): chunk_block_rsv: size 0 reserved 0
[117261.795127] BTRFS info (device sdb3): remap_block_rsv: size 0 reserved 0
[117261.802251] BTRFS info (device sdb3): delayed_block_rsv: size 0 reserved 0
[117261.809551] BTRFS info (device sdb3): delayed_refs_rsv: size 0 reserved 0
[117261.816785] BTRFS info (device sdb3): block group 55856594944 has 1073741824 bytes, 1007976448 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (65765376
 bytes available) [readonly]
[117261.834327] BTRFS info (device sdb3): block group has cluster?: no
[117261.840947] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes
[117261.849500] BTRFS info (device sdb3): block group 101368987648 has 1073741824 bytes, 500957184 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (57278464
0 bytes available) [readonly]
[117261.867163] BTRFS info (device sdb3): block group has cluster?: no
[117261.873784] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes [117261.882313] BTRFS info (device sdb3): block group 102442729472 has 1073741824 bytes, 633536512 used 0 pinned 0 reserved 0 delalloc 0 super 0 zone_unusable (44020531
2 bytes available) [readonly]
[117261.899994] BTRFS info (device sdb3): block group has cluster?: no
[117261.906630] BTRFS info (device sdb3): 0 free space entries at or bigger than 16384 bytes
[117261.915176] BTRFS info (device sdb3): 1078755328 bytes available across all block groups
[117261.923750] BTRFS warning (device sdb3): Skipping commit of aborted transaction.
[117261.931608] ------------[ cut here ]------------
[117261.936687] BTRFS: Transaction aborted (error -28)
[117261.941950] WARNING: fs/btrfs/transaction.c:2060 at btrfs_commit_transaction+0xaaa/0xe20 [btrfs], CPU#6: mount/2008
[117261.952905] Modules linked in: ntfs3 btrfs libblake2b raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod auto
fs4 hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid sr_mod cdrom uas usb_storage i915 drm_buddy ttm i2c_algo_bit drm_display_helper cec sd_mod rc_core drm_cli
ent_lib drm_kms_helper iTCO_wdt xhci_pci ahci intel_pmc_bxt drm libahci watchdog xhci_hcd firewire_ohci libata firewire_core r8169 video realtek crc_itu_t usbcore scsi_
mod fan i2c_i801 wmi button usb_common i2c_smbus scsi_common efivarfs
[117262.004021] CPU: 6 UID: 0 PID: 2008 Comm: mount Tainted: G W           7.0.10+deb14-amd64 #1 PREEMPT(lazy)  Debian 7.0.10-1
[117262.016343] Tainted: [W]=WARN
[117262.019853] Hardware name: LENOVO 10M8S14C42/3102, BIOS M16KT71A 09/14/2023
[117262.027348] RIP: 0010:btrfs_commit_transaction+0xaac/0xe20 [btrfs]
[117262.034117] Code: ff 48 8b 7c 24 08 31 c9 48 8d b7 48 01 00 00 e8 9a 7c 0b 00 48 c7 43 18 00 00 00 00 e9 73 fe ff ff 48 8d 3d 66 fd 34 00 89 ee <67> 48 0f b9 3a e9
f8 fe ff ff 49 8b bc 24 48 03 00 00 be 02 00 00
[117262.053426] RSP: 0018:ffffce9fc8b57418 EFLAGS: 00010292
[117262.059233] RAX: 0000000000000002 RBX: ffff8ac4d181a738 RCX: 0000000000000003
[117262.066946] RDX: 0000000002040001 RSI: 00000000ffffffe4 RDI: ffffffffc1028080
[117262.074658] RBP: 00000000ffffffe4 R08: 0000000000000000 R09: ffffce9fc8b57200
[117262.082364] R10: ffff8acbe23fff90 R11: 00000000fff7ffff R12: ffff8ac545885000
[117262.090075] R13: ffff8ac4c242ea00 R14: ffff8ac5458854c0 R15: ffff8ac4cff01b80
[117262.097779] FS:  00007f22c28cf940(0000) GS:ffff8acc3a4fc000(0000) knlGS:0000000000000000
[117262.106429] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[117262.112740] CR2: 00007f22c2cdfdd0 CR3: 00000001867ac001 CR4: 00000000003706f0
[117262.120434] Call Trace:
[117262.123451]  <TASK>
[117262.126121]  ? start_transaction+0x228/0x880 [btrfs]
[117262.131676]  flush_space+0x3c3/0x5a0 [btrfs]
[117262.136574]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[117262.142855]  ? calc_available_free_space.isra.0+0x70/0xc0 [btrfs]
[117262.149597]  priority_reclaim_metadata_space+0x96/0x130 [btrfs]
[117262.156132]  handle_reserve_ticket+0x5b/0x290 [btrfs]
[117262.161800]  ? btrfs_reduce_alloc_profile+0x5a/0x1d0 [btrfs]
[117262.168076]  reserve_bytes+0x268/0x4c0 [btrfs] [117262.173127] btrfs_reserve_metadata_bytes+0x1a/0xe0 [btrfs]
[117262.179306]  btrfs_block_rsv_refill+0x60/0x90 [btrfs]
[117262.184965]  evict_refill_and_join+0x47/0xe0 [btrfs]
[117262.190531]  btrfs_evict_inode+0x332/0x400 [btrfs]
[117262.195922]  evict+0xfd/0x270
[117262.199441]  btrfs_orphan_cleanup+0x21f/0x390 [btrfs]
[117262.205085]  btrfs_start_pre_rw_mount+0x2a0/0x6c0 [btrfs]
[117262.211071]  open_ctree+0x12fb/0x1730 [btrfs]
[117262.216028]  btrfs_get_tree.cold+0xb/0xb6 [btrfs]
[117262.221324]  ? security_inode_permission+0x4a/0x110
[117262.226747]  vfs_get_tree+0x29/0xd0
[117262.230776]  vfs_cmd_create+0x57/0xd0
[117262.234947]  __do_sys_fsconfig+0x2fc/0x5e0
[117262.239581]  do_syscall_64+0xcd/0x15f0
[117262.243873]  ? __fs_parse+0x68/0x1c0
[117262.247989]  ? aa_file_perm+0x18a/0x550
[117262.252368]  ? btrfs_parse_param+0x4f/0x960 [btrfs]
[117262.257818]  ? apparmor_file_permission+0x44/0xb0
[117262.263052]  ? fscontext_read+0x171/0x190
[117262.267594]  ? rw_verify_area+0x56/0x180
[117262.272039]  ? vfs_read+0xbf/0x390
[117262.275964]  ? __fs_parse+0x68/0x1c0
[117262.280063]  ? aa_file_perm+0x18a/0x550
[117262.284422]  ? btrfs_parse_param+0x4f/0x960 [btrfs]
[117262.289858]  ? apparmor_file_permission+0x44/0xb0
[117262.295082]  ? fscontext_read+0x171/0x190
[117262.299605]  ? rw_verify_area+0x56/0x180
[117262.304036]  ? vfs_read+0xbf/0x390
[117262.307943]  ? do_syscall_64+0x10a/0x15f0
[117262.312451]  ? from_kgid_munged+0x17/0x30
[117262.316955]  ? __do_sys_getgid+0x27/0x30
[117262.321369]  ? do_syscall_64+0x10a/0x15f0
[117262.325856]  ? from_kgid_munged+0x17/0x30
[117262.330340]  ? ksys_read+0x73/0xf0
[117262.334213]  ? do_syscall_64+0x10a/0x15f0
[117262.338694]  ? irqentry_exit+0x78/0x680
[117262.343000]  ? exc_page_fault+0x82/0x1d0
[117262.347384]  ? __irq_exit_rcu+0x3d/0xe0 [117262.351683] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[117262.357197] RIP: 0033:0x7f22c2d0eb4a
[117262.361218] Code: 73 01 c3 48 8b 0d c6 32 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 49 89 ca b8 af 01 00 00 0f 05 <48> 3d 01 f0 ff ff
73 01 c3 48 8b 0d 96 32 0d 00 f7 d8 64 89 01 48
[117262.380431] RSP: 002b:00007ffd5d547f68 EFLAGS: 00000246 ORIG_RAX: 00000000000001af
[117262.388464] RAX: ffffffffffffffda RBX: 0000557471e4da50 RCX: 00007f22c2d0eb4a
[117262.396057] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000003
[117262.403650] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
[117262.411240] R10: 0000000000000000 R11: 0000000000000246 R12: 0000557471e4ec40
[117262.418831] R13: 00007f22c2ea127c R14: 00007f22c2e9f460 R15: 00007f22c28cf7f0
[117262.426410]  </TASK>
[117262.429024] ---[ end trace 0000000000000000 ]---
[117262.434074] BTRFS info (device sdb3 state A): dumping space info:
[117262.440595] BTRFS info (device sdb3 state A): space_info DATA (sub-group id 0) has 17422925824 free, is not full
[117262.451184] BTRFS info (device sdb3 state A): space_info total=70262980608, used=52839989248, pinned=0, reserved=0, may_use=0, readonly=65536 zone_unusable=0
[117262.465659] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.476171] BTRFS info (device sdb3 state A): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.491598] BTRFS info (device sdb3 state A): space_info SYSTEM (sub-group id 0) has 0 free, is not full
[117262.501477] BTRFS info (device sdb3 state A): space_info total=33554432, used=16384, pinned=0, reserved=0, may_use=0, readonly=33538048 zone_unusable=0
[117262.515439] BTRFS info (device sdb3 state A): global_block_rsv: size 190382080 reserved 190382080
[117262.524731] BTRFS info (device sdb3 state A): trans_block_rsv: size 0 reserved 0
[117262.532538] BTRFS info (device sdb3 state A): chunk_block_rsv: size 0 reserved 0
[117262.540337] BTRFS info (device sdb3 state A): remap_block_rsv: size 0 reserved 0
[117262.548136] BTRFS info (device sdb3 state A): delayed_block_rsv: size 0 reserved 0
[117262.556110] BTRFS info (device sdb3 state A): delayed_refs_rsv: size 0 reserved 0
[117262.563998] BTRFS: error (device sdb3 state A) in cleanup_transaction:2060: errno=-28 No space left
[117262.573453] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.584033] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.599563] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080
[117262.608943] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0
[117262.616846] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0
[117262.624749] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0
[117262.632643] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0
[117262.640715] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0
[117262.648696] BTRFS info (device sdb3 state EA): space_info METADATA (sub-group id 0) has -190382080 free, is full
[117262.659312] BTRFS info (device sdb3 state EA): space_info total=3221225472, used=2142470144, pinned=0, reserved=0, may_use=190382080, readonly=1078755328 zone_unusable=0
[117262.674875] BTRFS info (device sdb3 state EA): global_block_rsv: size 190382080 reserved 190382080
[117262.684287] BTRFS info (device sdb3 state EA): trans_block_rsv: size 0 reserved 0
[117262.692225] BTRFS info (device sdb3 state EA): chunk_block_rsv: size 0 reserved 0
[117262.700155] BTRFS info (device sdb3 state EA): remap_block_rsv: size 0 reserved 0
[117262.708086] BTRFS info (device sdb3 state EA): delayed_block_rsv: size 0 reserved 0
[117262.716191] BTRFS info (device sdb3 state EA): delayed_refs_rsv: size 0 reserved 0
[117262.724260] BTRFS warning (device sdb3 state EA): could not allocate space for delete; will truncate on mount
[117262.734622] BTRFS error (device sdb3 state EA): Error removing orphan entry, stopping orphan cleanup
[117262.744220] BTRFS error (device sdb3 state EA): could not do orphan cleanup -28
[117262.752021] BTRFS error (device sdb3 state EA): commit super block returned -30
[117262.760739] BTRFS error (device sdb3 state EA): open_ctree failed: 
-28
mount: /var: fsconfig() failed: No space left on device.
       dmesg(1) may have more information after failed mount system call.
Quoted 3 lines

If the first command works but the second command does not, you could insert a btrfs device add command to add a partition. I'm not such a big fan of adding USB sticks or zram devices because if anything goes wrong, the file system ends up broken.

This is a bit of a heavy hammer to fix this problem but it might actually be faster than troubleshooting Btrfs enospc:

I am all for troubleshooting this enospc issue because it is the first time I cannot even add a device to fix the enospc... in fact an hour before this issue I had an enospc issue this time with the root partition that had very few space left, and adding a device worked well and it was pretty easy to fix. But here I have gigabytes of unused data and metadata, I am puzzled.

I was switching my SSD btrfs partitions (I plan to merge them all but for historical reason they are split as separate btrfs partitions instead of subvolumes of a single btrfs partition) from single to DUP, the new default.

 I believe the main issue here is that the conversion attempted to use until the last megabyte of metadata before errorring out, preventing any simple rescue operation afterwards, and even preventing from balancing data to free chunks for metadata.

It would have been fine for the conversion to error out and let me balance data then resume the conversion. The main issue is that the conversion used so much metadata space that I cannot even mount the btrfs partition to balance its data space.

Maybe the metadata conversion should guard/ straight error out when there is not enough space in metadata (here 3GB to duplicate 1,59GB does not fit) and there is no way to get a new free chunk for metadata and tell to free chunks first ?

Quoted 6 lines

It should still be possible to make the troubled Btrfs a read-only seed device using btrfs-tune; then mount it ro; use btrfs dev add using a device that's equal to or bigger; remount rw; then btrfs dev rem the 1st device, i.e. the ro seed. This will replicate/clone the file system from device 1 to device 2. And it'll take a while. But in the end you'll have a balanced file system on device 2.

It isn't strictly necessary for device 2 to be equal to or bigger. It is possible to add a smaller device, but before you remove the 1st device, you wlil need to delete files. The bigger files you choose to delete, the fewer you will need to delete to get to the underprovisioned 2nd device size. Then remove the 1st device. Now only the files that have not been deleted will have extents replicated to the 2nd device.

At the very least the seed-device is not being modified and shouldn't get any worse than it is. Chris Murphy

So with I tried with what I understood:
with a copy of the image I did of the partition: /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img

# btrfstune -S 1 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: please finish/cancel the running replace/balance before running this command

I tried:

# btrfs check --clear-space-cache v2 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
Opening filesystem to check...
Checking filesystem on /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
UUID: 13af326c-631f-482b-9c34-b59b4f100608
WARNING: --clear-space-cache option is deprecated, please use "btrfs rescue clear-space-cache" instead
ERROR: please finish/cancel the running replace/balance before running this command
# btrfs balance cancel /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: not a directory: /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
# btrfs rescue clear-uuid-tree /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: please finish/cancel the running replace/balance before running this command

and without the seed flag:

# dd if=/dev/zero of=/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var_target.img bs=1M count=90000
90000+0 records in
90000+0 records out
94371840000 bytes (94 GB, 88 GiB) copied, 1340.52 s, 70.4 MB/s
# losetup -f --show /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
[475088.702700] loop0: detected capacity change from 0 to 166119424
/dev/loop0
# mkdir -p /mnt/seed_workspace
# mount -o ro,rescue=all /dev/loop0 /mnt/seed_workspace
[475287.536430] BTRFS info: device /dev/loop0 (7:0) using temp-fsid 7d3aa685-b9cd-4cbc-87ed-c2d0288fdeba
[475287.546680] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/loop0 (7:0) scanned by mount (8576)
[475287.556933] BTRFS info (device loop0 state S): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[475287.567325] BTRFS info (device loop0 state S): using crc32c checksum algorithm
[475288.554301] BTRFS info (device loop0 state ECS): enabling ssd optimizations
[475288.562226] BTRFS info (device loop0 state ECS): disabling log replay at mount time
[475288.570394] BTRFS info (device loop0 state ECS): turning on async discard
[475288.577648] BTRFS info (device loop0 state ECS): enabling free space tree
[475288.584915] BTRFS info (device loop0 state ECS): ignoring bad roots
[475288.591652] BTRFS info (device loop0 state ECS): ignoring data csums
[475288.598482] BTRFS info (device loop0 state ECS): ignoring meta csums
[475288.605308] BTRFS info (device loop0 state ECS): ignoring unknown super block flags
# losetup -f --show /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var_target.img
[475310.285452] loop1: detected capacity change from 0 to 184320000
/dev/loop1
# btrfs device add /dev/loop1 /mnt/seed_workspace
Performing full device TRIM /dev/loop1 (87.89GiB) ...
ERROR: error adding device '/dev/loop1': Read-only file system
Quoted 6 lines
Quoted 6 lines

I don't know yet how to proceed with such a read-only seed option, I will look into it. But is there other flags or tune I could do that could let me cancel the pause operation, the balance the data space, then retry the conversion ?

Alban

Alban

#6 lore raw
Quoted 7 lines · Alban Browaeys

Hi,

I had space after the sdb3 btrfs /var partition, a 9GB swap partition. I deleted this swap partition and expanded the sdb3 btrfs size with parted But I don't seem to be able to expand the btrfs filesystem inside sdb3 if I cannot mount it read write, is it so? Or is it only due to my pending operation?

If the file system goes read-only it cannot be modified at all, including resize.

Quoted 1 line

Can I get this btrfs partition back?

I think there's a bug. I do know know if the bug wrote confusion to disk, but either way the current kernel code can't handle it so it's a bug. One of your dmesg contained this

Quoted 1 line
[ 1084.779326] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full

Can you run 'btrfs check --readonly' on this file system?

My recommendation is to stop mounting it rw. Only mount it ro. The more it is changed the worse the problem is likely to get.

Quoted 2 lines

Is there a way to copy the content of the partition mounted in ro in its current state, create a new partition and copy the content back to it?

mount -o ro

Then use rsync -a or cp -a

I think that's safest.

The other option I was suggesting might make it possible to fix, but I've never been able to try it in a case like what you're experiencing so it's entirely untested and therefore risky.

btrfstune -S1

This *changes the file system* therefore it's a risk. But I think it's a minimal change to just the superblock to make the file system a read-only seed device. This is the only exception to the rule that you cannot add a device when a Btrfs is read-only. It is possible to add a device to a seed device. But the unintuitive part is how to make it read-write.

btrfstune -S1 $device1
mount $device1 /mnt
btrfs device add $device2 /mnt
mount -o remount,rw /mnt

This is now a two device Btrfs file system. The seed device remains read-only. The second device receives all the writes/changes as a feature of COW. In effect it's a kind of overlay.

What I'm suggesting is if you now do:

btrfs device remove $device1 /mnt

This tells the btrfs kernel code to replicate the contents of device1 onto device2. This command will not complete until the replication is complete, which could take hours (no idea, depends on how full the file system is).

$device2 size needs to be at least as much as the used amount for $device1.

Quoted 4 lines

There are docker subvolumes on it wich seems they can be recreated by docker later on (I hope so) so rsync might be an option but if the docker subvolume need to be backed up rsync is not able to backup them correctly.

I do not know to what degree the docker btrfs graph driver uses reflinks. Using rsync or cp might dramatically explode the amount of storage needed. It could be docker usage is part of the problem if the problem is at all related to bookend extents. I seem to recall that docker graph driver does result in substantial amounts of bookend extents.

Quoted 1 line

And btrfs restore gives me thousands of

Btrfs restore may not help. It's a scraping tool. It's designed to get data out at all costs, even permitting extraction of corrupt files. It won't prefer reflinks or snapshots.

Quoted 4 lines

Also should I expect this behavior to happen everytime I do a btrfs filessytem conversion without checking I have enough metadata space available beforehand or will this be prevented by some new code one day ?

This is always the hard part about file systems. Computers are ordinarily deterministic. But file systems are increasingly non-deterministic as they age. And once you hit the bug, the state of the file system has already changed making it important to stop making all changes to the file system in order to preserve that state for file system developers. The more we hammer on trying to fix it, it's like a crime scene being cleaned up or tampered with. It makes it impossible for the developers to understand what happened and therefore how to fix it so it doesn't happen again.

Signature · 1 line
Chris Murphy
#7 lore raw

Hi Chris, Hi All,

There is still a thing I wonder, reagarding the size of the used metadata, and knowing that small data are stored on the metadata partition, could it be possible to force btrfs to move these small data items to the data partition that has 16GB free?
I doubt I have 1.5GB of pure metadata for around 50GB of data, even if there are a lot of small files.

Le 15/06/2026 à 04:07, Chris Murphy a écrit :

Quoted 14 lines · Alban Browaeys
Quoted 7 lines

Hi,

I had space after the sdb3 btrfs /var partition, a 9GB swap partition. I deleted this swap partition and expanded the sdb3 btrfs size with parted But I don't seem to be able to expand the btrfs filesystem inside sdb3 if I cannot mount it read write, is it so? Or is it only due to my pending operation?

If the file system goes read-only it cannot be modified at all, including resize.

Quoted 8 lines
Quoted 1 line

Can I get this btrfs partition back?

I think there's a bug. I do know know if the bug wrote confusion to disk, but either way the current kernel code can't handle it so it's a bug. One of your dmesg contained this

Quoted 1 line
[ 1084.779326] BTRFS info (device sdb3 state A): space_info METADATA (sub-group id 0) has -190382080 free, is full

Can you run 'btrfs check --readonly' on this file system?

 # btrfs check --readonly /dev/sdb3
 Opening filesystem to check...
 Checking filesystem on /dev/sdb3
 UUID: 13af326c-631f-482b-9c34-b59b4f100608
 [1/8] checking log skipped (none written)
 [2/8] checking root items
 [3/8] checking extents
 [4/8] checking free space tree
 [5/8] checking fs roots
 [6/8] checking only csums items (without verifying data)
 [7/8] checking root refs
 [8/8] checking quota groups skipped (not enabled on this FS)
 found 54982475776 bytes used, no error found
 total csum bytes: 50964248
 total tree bytes: 2142486528
 total fs tree bytes: 1951760384
 total extent tree bytes: 97910784
 btree space waste bytes: 569359506
 file data blocks allocated: 105614794752
  referenced 100030590976
Quoted 1 line

My recommendation is to stop mounting it rw. Only mount it ro. The more it is changed the worse the problem is likely to get.

So I did. I don't know if the previous attempt to mount it read write with default options that filed with ENOSPC did damages. But I have since refrained from attempting to mount it rw with any rescue option, I only mounted ro with the rescue options.

Quoted 11 lines
Quoted 4 lines

Is there a way to copy the content of the partition mounted in ro in its current state, create a new partition and copy the content back to it?

mount -o ro

Then use rsync -a or cp -a

I think that's safest.

Thank you, the doubt I had with rsync where about it flatting
subvolumes. I don't know a tool that does what rsync does without
flattening subvolumes. Thankfully the only subvolumes on this
partitions are /var/li/docker/btrfs that Gemini told me would be
recreated at startup by the docker dameon, so it told me to exclude
them from the rsync. For now the only reliable backup I am confident is
complete is the dd raw image... but it suffers from the same issue the
raw device partition has, ie that it is likely in a possibly
recoverable corrupt state (extends halfway in migration). Gemini told
me to try btrfs-restore but it gave me thousands of errors like:
"
ERROR: zstd frame incomplete
ERROR: copying data for /mnt/4/hermes-var-20260610-enospc-vanilla/restored_var/@var/backups/dpkg.arch.5.gz failed "

Quoted 11 lines

The other option I was suggesting might make it possible to fix, but I've never been able to try it in a case like what you're experiencing so it's entirely untested and therefore risky.

btrfstune -S1

This *changes the file system* therefore it's a risk. But I think it's a minimal change to just the superblock to make the file system a read-only seed device. This is the only exception to the rule that you cannot add a device when a Btrfs is read-only. It is possible to add a device to a seed device. But the unintuitive part is how to make it read-write.

btrfstune -S1 $device1
mount $device1 /mnt
btrfs device add $device2 /mnt
mount -o remount,rw /mnt

Sadly I tried it and reported about it in my previous email (but my bad I should have shortened the bug citation block in the middle which probably lead you to conclude there was only citation until the end of the email). The issue is I cannot mark the filesystem as seed if an operation is pending. (and as seen before I cannot cancel this operation it seems if I cannot mount the filesystem read write, dead lock). Here it is:

So with I tried with what I understood:
with a copy of the image I did of the partition:
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img

# btrfstune -S 1 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: please finish/cancel the running replace/balance before running this command

I tried:

# btrfs check --clear-space-cache v2 /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
Opening filesystem to check...
Checking filesystem on /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
UUID: 13af326c-631f-482b-9c34-b59b4f100608
WARNING: --clear-space-cache option is deprecated, please use "btrfs rescue clear-space-cache" instead
ERROR: please finish/cancel the running replace/balance before running this command
# btrfs balance cancel /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: not a directory: /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
# btrfs rescue clear-uuid-tree /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: please finish/cancel the running replace/balance before running this command

and without the seed flag:

# dd if=/dev/zero of=/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var_target.img bs=1M count=90000
90000+0 records in
90000+0 records out
94371840000 bytes (94 GB, 88 GiB) copied, 1340.52 s, 70.4 MB/s
# losetup -f --show /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
[475088.702700] loop0: detected capacity change from 0 to 166119424
/dev/loop0
 # mkdir -p /mnt/seed_workspace
# mount -o ro,rescue=all /dev/loop0 /mnt/seed_workspace
[475287.536430] BTRFS info: device /dev/loop0 (7:0) using temp-fsid 7d3aa685-b9cd-4cbc-87ed-c2d0288fdeba
[475287.546680] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/loop0 (7:0) scanned by mount (8576)
[475287.556933] BTRFS info (device loop0 state S): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[475287.567325] BTRFS info (device loop0 state S): using crc32c checksum algorithm
[475288.554301] BTRFS info (device loop0 state ECS): enabling ssd optimizations
[475288.562226] BTRFS info (device loop0 state ECS): disabling log replay at mount time
[475288.570394] BTRFS info (device loop0 state ECS): turning on async discard
[475288.577648] BTRFS info (device loop0 state ECS): enabling free space tree
[475288.584915] BTRFS info (device loop0 state ECS): ignoring bad roots
[475288.591652] BTRFS info (device loop0 state ECS): ignoring data csums
[475288.598482] BTRFS info (device loop0 state ECS): ignoring meta csums
[475288.605308] BTRFS info (device loop0 state ECS): ignoring unknown super block flags
# losetup -f --show /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var_target.img
[475310.285452] loop1: detected capacity change from 0 to 184320000
/dev/loop1
# btrfs device add /dev/loop1 /mnt/seed_workspace
Performing full device TRIM /dev/loop1 (87.89GiB) ...
[475344.936908] BTRFS error (device loop0 state ECS): device add not supported on cloned temp-fsid mount
ERROR: error adding device '/dev/loop1': Invalid argument
Quoted 15 lines

This is now a two device Btrfs file system. The seed device remains read-only. The second device receives all the writes/changes as a feature of COW. In effect it's a kind of overlay.

What I'm suggesting is if you now do:

btrfs device remove $device1 /mnt

This tells the btrfs kernel code to replicate the contents of device1 onto device2. This command will not complete until the replication is complete, which could take hours (no idea, depends on how full the file system is).

$device2 size needs to be at least as much as the used amount for $device1.

Quoted 4 lines

There are docker subvolumes on it wich seems they can be recreated by docker later on (I hope so) so rsync might be an option but if the docker subvolume need to be backed up rsync is not able to backup them correctly.

Quoted 6 lines

I do not know to what degree the docker btrfs graph driver uses reflinks. Using rsync or cp might dramatically explode the amount of storage needed. It could be docker usage is part of the problem if the problem is at all related to bookend extents. I seem to recall that docker graph driver does result in substantial amounts of bookend extents.

Quoted 1 line

And btrfs restore gives me thousands of

Btrfs restore may not help. It's a scraping tool. It's designed to get data out at all costs, even permitting extraction of corrupt files. It won't prefer reflinks or snapshots.

Ok. Though I believe that if btrfs-restore find incomplete zstd frame it might be that rsync will not be able to copy everything from the ro half converted btrfs partition?

Quoted 6 lines
Quoted 4 lines

Also should I expect this behavior to happen everytime I do a btrfs filessytem conversion without checking I have enough metadata space available beforehand or will this be prevented by some new code one day ?

This is always the hard part about file systems. Computers are ordinarily deterministic. But file systems are increasingly non-deterministic as they age. And once you hit the bug, the state of the file system has already changed making it important to stop making all changes to the file system in order to preserve that state for file system developers. The more we hammer on trying to fix it, it's like a crime scene being cleaned up or tampered with. It makes it impossible for the developers to understand what happened and therefore how to fix it so it doesn't happen again.

Sure. Out of plain "mount /dev/sdb3 /var" I haven't done any rw mount attempts. I hope these attempts did not make the issue worst.

Alban

#8 lore raw
Quoted 7 lines · Alban Browaeys

Hi Chris, Hi All,

There is still a thing I wonder, reagarding the size of the used metadata, and knowing that small data are stored on the metadata partition, could it be possible to force btrfs to move these small data items to the data partition that has 16GB free?

No. By default Btrfs has separate block group types so it's not possible for Btrfs to put metadata into the unused space in data block groups or vice versa. That is permitted with mixed block group type.

Quoted 12 lines
 # btrfs check --readonly /dev/sdb3
 Opening filesystem to check...
 Checking filesystem on /dev/sdb3
 UUID: 13af326c-631f-482b-9c34-b59b4f100608
 [1/8] checking log skipped (none written)
 [2/8] checking root items
 [3/8] checking extents
 [4/8] checking free space tree
 [5/8] checking fs roots
 [6/8] checking only csums items (without verifying data)
 [7/8] checking root refs
 [8/8] checking quota groups skipped (not enabled on this FS)
Quoted 8 lines
 found 54982475776 bytes used, no error found
 total csum bytes: 50964248
 total tree bytes: 2142486528
 total fs tree bytes: 1951760384
 total extent tree bytes: 97910784
 btree space waste bytes: 569359506
 file data blocks allocated: 105614794752
  referenced 100030590976

OK so the file system is consistent. And there's no log tree.

For some reason skip_balance is not able to abort the balance/convert before the file system runs into the problem and goes read-only. Read-only prevents us from doing everything else.

Clearly the file system wants to allocate at least one metadata block group but cannot, there's no unallocated space on the device and no way to resize or add more space.

At the moment it appears to be stuck. So it's a bug somewhere.

Quoted 5 lines

Here it is:

So with I tried with what I understood:
with a copy of the image I did of the partition:
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img

Quoted 7 lines
# btrfstune -S 1 
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: please finish/cancel the running replace/balance before running 
this command

I tried:

# btrfs check --clear-space-cache v2 

Yeah I advise not removing either version of space cache until the balance is cancelled. But we can't cancel the balance before the file system goes read-only which mean the balance cannot be cancelled. This is unrelated to the free space tree.

But yeah too bad the seed flag cannot be set on such a file system.

Quoted 15 lines

/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img Opening filesystem to check...
Checking filesystem on /mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
UUID: 13af326c-631f-482b-9c34-b59b4f100608
WARNING: --clear-space-cache option is deprecated, please use "btrfs rescue clear-space-cache" instead
ERROR: please finish/cancel the running replace/balance before running this command

# btrfs balance cancel 
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: not a directory: 
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
Quoted 4 lines
# btrfs rescue clear-uuid-tree 
/mnt/4/hermes-var-20260610-enospc-vanilla/tests/sdb3_var.img
ERROR: please finish/cancel the running replace/balance before running 
this command

Clearing trees while a balance is in progress is not recommended. So it's a good thing the tool doesn't permit it.

Quoted 3 lines

Ok. Though I believe that if btrfs-restore find incomplete zstd frame it might be that rsync will not be able to copy everything from the ro half converted btrfs partition?

I don't know what the btrfs restore complaint about zstd frames is about. That makes me think there is corrupt data blocks, and the block can't be decompressed by zstd.

You could try:

mount -o ro $dev1 /mnt
btrfs scrub start -Brd /mnt

That will do a read-only scrub on the read-only file system. It can't fix anything but it will show if there's corrupted files.

In any case, the file system does mount ro normally. You can reliably get important information out. The most portable backup/restore for podman/docker/mobi is tar - I know they have internal backup commands that use tar but I do not know what options are actually used and therefore how to make a tar backup of your containers rather than relying on rsync - it's not my area. But I would investigate a more generic way of getting important containers out of the broken file system so you can preserve them no matter the file system or graph driver you decide to use.

Signature · 1 line
Chris Murphy
#9 lore raw

Have you seen I posted a link to a btrfs debug image in a previous email in this thread?

"
partition sdb3_var_metadata-only-debug.img at https://drive.google.com/file/d/1OzEZLaTUNpWsL1ohGKOMW4GiR9oQakdP/view?usp=drive_link done with "btrfs-image -c 9 -t 4 /dev/sdb3 /mnt/4/hermes-var-20260610-enospc-vanilla/sdb3_var_metadata-only-debug.img". "

I made an image and a btrfs debug image, is there any use for me to keep the broken on disk partition? I believe I cannot expect to be able to recover this on disk partition in a week and thus should delete it and recreate it if I want the pc back? And keep the disk partition image for debugging.

Le 15/06/2026 à 22:00, Chris Murphy a écrit :

Quoted 37 lines · Alban Browaeys
Quoted 13 lines
 # btrfs check --readonly /dev/sdb3
 Opening filesystem to check...
 Checking filesystem on /dev/sdb3
 UUID: 13af326c-631f-482b-9c34-b59b4f100608
 [1/8] checking log skipped (none written)
 [2/8] checking root items
 [3/8] checking extents
 [4/8] checking free space tree
 [5/8] checking fs roots
 [6/8] checking only csums items (without verifying data)
 [7/8] checking root refs
 [8/8] checking quota groups skipped (not enabled on this FS)
Quoted 8 lines
 found 54982475776 bytes used, no error found
 total csum bytes: 50964248
 total tree bytes: 2142486528
 total fs tree bytes: 1951760384
 total extent tree bytes: 97910784
 btree space waste bytes: 569359506
 file data blocks allocated: 105614794752
  referenced 100030590976

OK so the file system is consistent. And there's no log tree.

For some reason skip_balance is not able to abort the balance/convert before the file system runs into the problem and goes read-only. Read-only prevents us from doing everything else.

Clearly the file system wants to allocate at least one metadata block group but cannot, there's no unallocated space on the device and no way to resize or add more space.

At the moment it appears to be stuck. So it's a bug somewhere.

Quoted 5 lines
Quoted 3 lines

Ok. Though I believe that if btrfs-restore find incomplete zstd frame it might be that rsync will not be able to copy everything from the ro half converted btrfs partition?

I don't know what the btrfs restore complaint about zstd frames is about. That makes me think there is corrupt data blocks, and the block can't be decompressed by zstd.

Could it be half transfered data during the conversion ?

Quoted 4 lines

You could try:

mount -o ro $dev1 /mnt
btrfs scrub start -Brd /mnt

# mount -o ro /dev/sdb3 /var
[820430.153273] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (13961)
[820430.175471] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[820430.185087] BTRFS info (device sdb3): using crc32c checksum algorithm
[820430.221645] BTRFS info (device sdb3): enabling ssd optimizations
[820430.228181] BTRFS info (device sdb3): turning on async discard
[820430.234575] BTRFS info (device sdb3): enabling free space tree
# btrfs scrub start -Brd /var
WARNING: cannot create scrub data file, mkdir /v[820447.453110] BTRFS info (device sdb3): scrub: started on devid 1
ar/lib failed: Read-only file system, status recording disabled
WARNING: failed to open the progress status socket at /var/lib/btrfs/scrub.progress.13af326c-631f-482b-9c34-b59b4f100608: No such file or directory, progress cannot be queried
Starting scrub on devid 1
[820598.534072] BTRFS info (device sdb3): scrub: finished on devid 1 with status: 0

Scrub device /dev/sdb3 (id 1) done

Scrub started:    Fri Jun 19 01:51:24 2026
Status:           finished
Duration:         0:02:31
Total to scrub:   51.21GiB
Rate:             347.25MiB/s
Error summary:    no errors found
# btrfs fi df /var
Data, single: total=65.44GiB, used=49.21GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=3.00GiB, used=2.00GiB
GlobalReserve, single: total=181.56MiB, used=0.00B
Quoted 3 lines

That will do a read-only scrub on the read-only file system. It can't fix anything but it will show if there's corrupted files.

In any case, the file system does mount ro normally. You can reliably get important information out. The most portable backup/restore for podman/docker/mobi is tar - I know they have internal backup commands that use tar but I do not know what options are actually used and therefore how to make a tar backup of your containers rather than relying on rsync - it's not my area. But I would investigate a more generic way of getting important containers out of the broken file system so you can preserve them no matter the file system or graph driver you decide to use.

Alban