Skip to contentSkip to conversation

Filesystem Read Only due to errno=-28 during metadata allocation

DISCUSSION [email protected] 3 participants 7 messages Oct 13, 2021 – Oct 25, 2021 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hello,

I faced issue with btrfs FS /var forced to RO due to errno=-28 (no space left).

The server was restarted to bring back FS in RW.

Before reboot:

$ btrfs fi usage /var -m
Overall:
    Device size:         2560.00MiB
    Device allocated:         2559.00MiB
    Device unallocated:            1.00MiB
    Device missing:            0.00MiB
    Used:         1116.00MiB
    Free (estimated):          451.25MiB (min: 451.25MiB)
    Data ratio:               1.00
    Metadata ratio:               2.00
    Global reserve:           13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1108.00MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:467.88MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 935.75MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 1.00MiB

The FS went RO on Sunday, with this trace:

2021-10-10T00:13:12.790042+02:00 SERVERNAME kernel: BTRFS: Transaction aborted (error -28)
2021-10-10T00:13:12.790053+02:00 SERVERNAME kernel: ------------[ cut here ]------------
2021-10-10T00:13:12.790055+02:00 SERVERNAME kernel: WARNING: CPU: 8 PID: 8532 at ../fs/btrfs/extent-tree.c:2353 btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790057+02:00 SERVERNAME kernel: Modules linked in: lin_tape(OEX) pfo(OEX) nfsv3 nfs_acl nfs lockd grace sunrpc fscache rpadlpar_io(X) rpaphp(X) tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc af_packet xfs libcrc32c st ch ibmveth(X) vmx_crypto gf128mul crct10dif_vpmsum uio_pdrv_genirq uio rtc_generic btrfs xor zstd_decompress zstd_compress xxhash raid6_pq dm_service_time sd_mod ibmvfc(X) scsi_transport_fc crc32c_vpmsum dm_mirror dm_region_hash dm_log sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod autofs4
2021-10-10T00:13:12.790059+02:00 SERVERNAME kernel: Supported: Yes, External
2021-10-10T00:13:12.790053+02:00 SERVERNAME kernel: ------------[ cut here ]------------
2021-10-10T00:13:12.790055+02:00 SERVERNAME kernel: WARNING: CPU: 8 PID: 8532 at ../fs/btrfs/extent-tree.c:2353 btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790057+02:00 SERVERNAME kernel: Modules linked in: lin_tape(OEX) pfo(OEX) nfsv3 nfs_acl nfs lockd grace sunrpc fscache rpadlpar_io(X) rpaphp(X) tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc af_packet xfs libcrc32c st ch ibmveth(X) vmx_crypto gf128mul crct10dif_vpmsum uio_pdrv_genirq uio rtc_generic btrfs xor zstd_decompress zstd_compress xxhash raid6_pq dm_service_time sd_mod ibmvfc(X) scsi_transport_fc crc32c_vpmsum dm_mirror dm_region_hash dm_log sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod autofs4
2021-10-10T00:13:12.790059+02:00 SERVERNAME kernel: Supported: Yes, External
2021-10-10T00:13:12.790060+02:00 SERVERNAME kernel: CPU: 8 PID: 8532 Comm: vpdupdate Tainted: G W OE 4.12.14-122.83-default #1 SLE12-SP5
2021-10-10T00:13:12.790076+02:00 SERVERNAME kernel: task: c0000068d70c1600 task.stack: c000002aba1a0000
2021-10-10T00:13:12.790078+02:00 SERVERNAME kernel: NIP: d000000035876474 LR: d000000035876470 CTR: 0000000000000000
2021-10-10T00:13:12.790080+02:00 SERVERNAME kernel: REGS: c000002aba1a3870 TRAP: 0700 Tainted: G W OE (4.12.14-122.83-default)
2021-10-10T00:13:12.790081+02:00 SERVERNAME kernel: MSR: 800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>
2021-10-10T00:13:12.790083+02:00 SERVERNAME kernel: CR: 24444422 XER: 20040000
2021-10-10T00:13:12.790085+02:00 SERVERNAME kernel: CFAR: c0000000009c817c SOFTE: 1
2021-10-10T00:13:12.790086+02:00 SERVERNAME kernel: GPR00: d000000035876470 c000002aba1a3af0 d000000035993288 0000000000000026
2021-10-10T00:13:12.790088+02:00 SERVERNAME kernel: GPR04: c00000e6bf30ade8 c00000e6bf322a00 0000000000000007 c0000000013ed474
2021-10-10T00:13:12.790090+02:00 SERVERNAME kernel: GPR08: 0000000000000000 c000000000dc16fc 000000e6be550000 000000000000134f
2021-10-10T00:13:12.790091+02:00 SERVERNAME kernel: GPR12: 0000000000004000 c00000000f6c9400 00000000007fffff 0000000000000014
2021-10-10T00:13:12.790093+02:00 SERVERNAME kernel: GPR16: 00000100231aa080 0000010022868988 aaaaaaaaaaaaaaab c000000024ebf778
2021-10-10T00:13:12.790095+02:00 SERVERNAME kernel: GPR20: 0000000000000000 0000000000000000 0000000000000000 c00000e5a1f6eb30
2021-10-10T00:13:12.790097+02:00 SERVERNAME kernel: GPR24: c00000e5a1f6eb20 00000000000016ba c00000e5a1f6e9c0 c00000e6828cc000
2021-10-10T00:13:12.790098+02:00 SERVERNAME kernel: GPR28: c0000014b1bc02d0 0000000000000000 c00000e6828cc000 ffffffffffffffe4
2021-10-10T00:13:12.790100+02:00 SERVERNAME kernel: NIP [d000000035876474] btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790101+02:00 SERVERNAME kernel: LR [d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
2021-10-10T00:13:12.790103+02:00 SERVERNAME kernel: Call Trace:
2021-10-10T00:13:12.790104+02:00 SERVERNAME kernel: [c000002aba1a3af0] [d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs] (unreliable)
2021-10-10T00:13:12.790106+02:00 SERVERNAME kernel: [c000002aba1a3bb0] [d000000035891554] btrfs_commit_transaction+0x74/0xc10 [btrfs]
2021-10-10T00:13:12.790108+02:00 SERVERNAME kernel: [c000002aba1a3c80] [d0000000358b3328] btrfs_sync_file+0x3a8/0x510 [btrfs]
2021-10-10T00:13:12.790110+02:00 SERVERNAME kernel: [c000002aba1a3d80] [c000000000408720] vfs_fsync_range+0x70/0x120
2021-10-10T00:13:12.790111+02:00 SERVERNAME kernel: [c000002aba1a3dd0] [c00000000040886c] do_fsync+0x5c/0xb0
2021-10-10T00:13:12.790113+02:00 SERVERNAME kernel: [c000002aba1a3e10] [c000000000408cfc] SyS_fdatasync+0x2c/0x40
2021-10-10T00:13:12.790115+02:00 SERVERNAME kernel: [c000002aba1a3e30] [c00000000000b308] system_call+0x3c/0x130
2021-10-10T00:13:12.790116+02:00 SERVERNAME kernel: Instruction dump:
2021-10-10T00:13:12.790118+02:00 SERVERNAME kernel: e87c0060 3c820000 e8848b48 38a0fffb 4bfe6905 60000000 4bffffb4 3c620000
2021-10-10T00:13:12.790120+02:00 SERVERNAME kernel: e8638b38 7fe4fb78 480e3f55 e8410018 <0fe00000> 4bffff98 60000000 3c4c0012
2021-10-10T00:13:12.790122+02:00 SERVERNAME kernel: ---[ end trace c9aa23777165dfdc ]---
2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error (device dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12 callbacks suppressed
2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device dm-22): forced readonly
2021-10-10T00:13:12.790060+02:00 SERVERNAME kernel: CPU: 8 PID: 8532 Comm: vpdupdate Tainted: G W OE 4.12.14-122.83-default #1 SLE12-SP5
2021-10-10T00:13:12.790076+02:00 SERVERNAME kernel: task: c0000068d70c1600 task.stack: c000002aba1a0000
2021-10-10T00:13:12.790078+02:00 SERVERNAME kernel: NIP: d000000035876474 LR: d000000035876470 CTR: 0000000000000000
2021-10-10T00:13:12.790080+02:00 SERVERNAME kernel: REGS: c000002aba1a3870 TRAP: 0700 Tainted: G W OE (4.12.14-122.83-default)
2021-10-10T00:13:12.790081+02:00 SERVERNAME kernel: MSR: 800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>
2021-10-10T00:13:12.790083+02:00 SERVERNAME kernel: CR: 24444422 XER: 20040000
2021-10-10T00:13:12.790085+02:00 SERVERNAME kernel: CFAR: c0000000009c817c SOFTE: 1
2021-10-10T00:13:12.790086+02:00 SERVERNAME kernel: GPR00: d000000035876470 c000002aba1a3af0 d000000035993288 0000000000000026
2021-10-10T00:13:12.790088+02:00 SERVERNAME kernel: GPR04: c00000e6bf30ade8 c00000e6bf322a00 0000000000000007 c0000000013ed474
2021-10-10T00:13:12.790090+02:00 SERVERNAME kernel: GPR08: 0000000000000000 c000000000dc16fc 000000e6be550000 000000000000134f
2021-10-10T00:13:12.790091+02:00 SERVERNAME kernel: GPR12: 0000000000004000 c00000000f6c9400 00000000007fffff 0000000000000014
2021-10-10T00:13:12.790093+02:00 SERVERNAME kernel: GPR16: 00000100231aa080 0000010022868988 aaaaaaaaaaaaaaab c000000024ebf778
2021-10-10T00:13:12.790095+02:00 SERVERNAME kernel: GPR20: 0000000000000000 0000000000000000 0000000000000000 c00000e5a1f6eb30
2021-10-10T00:13:12.790097+02:00 SERVERNAME kernel: GPR24: c00000e5a1f6eb20 00000000000016ba c00000e5a1f6e9c0 c00000e6828cc000
2021-10-10T00:13:12.790098+02:00 SERVERNAME kernel: GPR28: c0000014b1bc02d0 0000000000000000 c00000e6828cc000 ffffffffffffffe4
2021-10-10T00:13:12.790100+02:00 SERVERNAME kernel: NIP [d000000035876474] btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790101+02:00 SERVERNAME kernel: LR [d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
2021-10-10T00:13:12.790103+02:00 SERVERNAME kernel: Call Trace:
2021-10-10T00:13:12.790104+02:00 SERVERNAME kernel: [c000002aba1a3af0] [d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs] (unreliable)
2021-10-10T00:13:12.790106+02:00 SERVERNAME kernel: [c000002aba1a3bb0] [d000000035891554] btrfs_commit_transaction+0x74/0xc10 [btrfs]
2021-10-10T00:13:12.790108+02:00 SERVERNAME kernel: [c000002aba1a3c80] [d0000000358b3328] btrfs_sync_file+0x3a8/0x510 [btrfs]
2021-10-10T00:13:12.790110+02:00 SERVERNAME kernel: [c000002aba1a3d80] [c000000000408720] vfs_fsync_range+0x70/0x120
2021-10-10T00:13:12.790111+02:00 SERVERNAME kernel: [c000002aba1a3dd0] [c00000000040886c] do_fsync+0x5c/0xb0
2021-10-10T00:13:12.790113+02:00 SERVERNAME kernel: [c000002aba1a3e10] [c000000000408cfc] SyS_fdatasync+0x2c/0x40
2021-10-10T00:13:12.790115+02:00 SERVERNAME kernel: [c000002aba1a3e30] [c00000000000b308] system_call+0x3c/0x130
2021-10-10T00:13:12.790116+02:00 SERVERNAME kernel: Instruction dump:
2021-10-10T00:13:12.790118+02:00 SERVERNAME kernel: e87c0060 3c820000 e8848b48 38a0fffb 4bfe6905 60000000 4bffffb4 3c620000
2021-10-10T00:13:12.790120+02:00 SERVERNAME kernel: e8638b38 7fe4fb78 480e3f55 e8410018 <0fe00000> 4bffff98 60000000 3c4c0012
2021-10-10T00:13:12.790122+02:00 SERVERNAME kernel: ---[ end trace c9aa23777165dfdc ]---
2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error (device dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12 callbacks suppressed
2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device dm-22): forced readonly
$ btrfs --version
btrfs-progs v4.5.3+20160729
$ btrfs fi show /var
Label: none  uuid: f96f4980-4682-4d2d-8d7a-3c0e2c1c6680
        Total devices 1 FS bytes used 1.06GiB
        devid    1 size 2.50GiB used 2.50GiB path 
/dev/mapper/rootvg-varvol

uname -a
Linux SERVERNAME 4.12.14-122.83-default #1 SMP Tue Aug 3 08:37:22 UTC 2021 (c86c48c) ppc64le ppc64le ppc64le GNU/Linux

On the previous Friday after weekly balance:
btrfs fi usage /var
Overall:
Device size: 2.50GiB
Device allocated: 1.73GiB
Device unallocated: 792.75MiB
Device missing: 0.00B
Used: 1.09GiB
Free (estimated): 1.11GiB (min: 739.62MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00B)

Data,single: Size:1.41GiB, Used:1.08GiB
/dev/mapper/rootvg-varvol 1.41GiB

Metadata,DUP: Size:128.00MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 256.00MiB

System,DUP: Size:32.00MiB, Used:64.00KiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 792.75MiB

I don't have extract of btrfs fi usage /var command during the weekend, but a script is extracting the Space allocated ("Size") and Used in Data and Metadata. I observed twice during the weekend space allocated to metadata is suddenly growing while the metadata used remains the same. The first time I had enough "Device unallocated" and no problem was observed, the second (on Sunday after midnight), it leads to FS RO (no space left).

Is there any situation that can lead to metadata allocation but without actual usage of metadata?

#1 lore raw
Quoted 238 lines · On 13.10.2021 14:35, [email protected] wrote:

Hello,

I faced issue with btrfs FS /var forced to RO due to errno=-28 (no space left).

The server was restarted to bring back FS in RW.

Before reboot:

$ btrfs fi usage /var -m
Overall:
    Device size:         2560.00MiB
    Device allocated:         2559.00MiB
    Device unallocated:            1.00MiB
    Device missing:            0.00MiB
    Used:         1116.00MiB
    Free (estimated):          451.25MiB (min: 451.25MiB)
    Data ratio:               1.00
    Metadata ratio:               2.00
    Global reserve:           13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1108.00MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:467.88MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 935.75MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 1.00MiB

The FS went RO on Sunday, with this trace:

2021-10-10T00:13:12.790042+02:00 SERVERNAME kernel: BTRFS: Transaction
aborted (error -28)
2021-10-10T00:13:12.790053+02:00 SERVERNAME kernel: ------------[ cut
here ]------------
2021-10-10T00:13:12.790055+02:00 SERVERNAME kernel: WARNING: CPU: 8
PID: 8532 at ../fs/btrfs/extent-tree.c:2353
btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790057+02:00 SERVERNAME kernel: Modules linked in:

lin_tape(OEX) pfo(OEX) nfsv3 nfs_acl nfs lockd grace sunrpc fscache rpadlpar_io(X) rpaphp(X) tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc af_packet xfs libcrc32c st ch ibmveth(X) vmx_crypto gf128mul crct10dif_vpmsum uio_pdrv_genirq uio rtc_generic btrfs xor zstd_decompress zstd_compress xxhash raid6_pq dm_service_time sd_mod ibmvfc(X) scsi_transport_fc crc32c_vpmsum dm_mirror dm_region_hash dm_log sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod autofs4

2021-10-10T00:13:12.790059+02:00 SERVERNAME kernel: Supported: Yes, External
2021-10-10T00:13:12.790053+02:00 SERVERNAME kernel: ------------[ cut
here ]------------
2021-10-10T00:13:12.790055+02:00 SERVERNAME kernel: WARNING: CPU: 8
PID: 8532 at ../fs/btrfs/extent-tree.c:2353
btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790057+02:00 SERVERNAME kernel: Modules linked in:

lin_tape(OEX) pfo(OEX) nfsv3 nfs_acl nfs lockd grace sunrpc fscache rpadlpar_io(X) rpaphp(X) tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc af_packet xfs libcrc32c st ch ibmveth(X) vmx_crypto gf128mul crct10dif_vpmsum uio_pdrv_genirq uio rtc_generic btrfs xor zstd_decompress zstd_compress xxhash raid6_pq dm_service_time sd_mod ibmvfc(X) scsi_transport_fc crc32c_vpmsum dm_mirror dm_region_hash dm_log sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod autofs4

2021-10-10T00:13:12.790059+02:00 SERVERNAME kernel: Supported: Yes, External
2021-10-10T00:13:12.790060+02:00 SERVERNAME kernel: CPU: 8 PID: 8532
Comm: vpdupdate Tainted: G W OE 4.12.14-122.83-default #1 SLE12-SP5
2021-10-10T00:13:12.790076+02:00 SERVERNAME kernel: task:
c0000068d70c1600 task.stack: c000002aba1a0000
2021-10-10T00:13:12.790078+02:00 SERVERNAME kernel: NIP:
d000000035876474 LR: d000000035876470 CTR: 0000000000000000
2021-10-10T00:13:12.790080+02:00 SERVERNAME kernel: REGS:
c000002aba1a3870 TRAP: 0700 Tainted: G W OE (4.12.14-122.83-default)
2021-10-10T00:13:12.790081+02:00 SERVERNAME kernel: MSR:
800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>
2021-10-10T00:13:12.790083+02:00 SERVERNAME kernel: CR: 24444422 XER: 20040000
2021-10-10T00:13:12.790085+02:00 SERVERNAME kernel: CFAR:
c0000000009c817c SOFTE: 1
2021-10-10T00:13:12.790086+02:00 SERVERNAME kernel: GPR00:
d000000035876470 c000002aba1a3af0 d000000035993288 0000000000000026
2021-10-10T00:13:12.790088+02:00 SERVERNAME kernel: GPR04:
c00000e6bf30ade8 c00000e6bf322a00 0000000000000007 c0000000013ed474
2021-10-10T00:13:12.790090+02:00 SERVERNAME kernel: GPR08:
0000000000000000 c000000000dc16fc 000000e6be550000 000000000000134f
2021-10-10T00:13:12.790091+02:00 SERVERNAME kernel: GPR12:
0000000000004000 c00000000f6c9400 00000000007fffff 0000000000000014
2021-10-10T00:13:12.790093+02:00 SERVERNAME kernel: GPR16:
00000100231aa080 0000010022868988 aaaaaaaaaaaaaaab c000000024ebf778
2021-10-10T00:13:12.790095+02:00 SERVERNAME kernel: GPR20:
0000000000000000 0000000000000000 0000000000000000 c00000e5a1f6eb30
2021-10-10T00:13:12.790097+02:00 SERVERNAME kernel: GPR24:
c00000e5a1f6eb20 00000000000016ba c00000e5a1f6e9c0 c00000e6828cc000
2021-10-10T00:13:12.790098+02:00 SERVERNAME kernel: GPR28:
c0000014b1bc02d0 0000000000000000 c00000e6828cc000 ffffffffffffffe4
2021-10-10T00:13:12.790100+02:00 SERVERNAME kernel: NIP
[d000000035876474] btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790101+02:00 SERVERNAME kernel: LR
[d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
2021-10-10T00:13:12.790103+02:00 SERVERNAME kernel: Call Trace:
2021-10-10T00:13:12.790104+02:00 SERVERNAME kernel: [c000002aba1a3af0]
[d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
(unreliable)
2021-10-10T00:13:12.790106+02:00 SERVERNAME kernel: [c000002aba1a3bb0]
[d000000035891554] btrfs_commit_transaction+0x74/0xc10 [btrfs]
2021-10-10T00:13:12.790108+02:00 SERVERNAME kernel: [c000002aba1a3c80]
[d0000000358b3328] btrfs_sync_file+0x3a8/0x510 [btrfs]
2021-10-10T00:13:12.790110+02:00 SERVERNAME kernel: [c000002aba1a3d80]
[c000000000408720] vfs_fsync_range+0x70/0x120
2021-10-10T00:13:12.790111+02:00 SERVERNAME kernel: [c000002aba1a3dd0]
[c00000000040886c] do_fsync+0x5c/0xb0
2021-10-10T00:13:12.790113+02:00 SERVERNAME kernel: [c000002aba1a3e10]
[c000000000408cfc] SyS_fdatasync+0x2c/0x40
2021-10-10T00:13:12.790115+02:00 SERVERNAME kernel: [c000002aba1a3e30]
[c00000000000b308] system_call+0x3c/0x130
2021-10-10T00:13:12.790116+02:00 SERVERNAME kernel: Instruction dump:
2021-10-10T00:13:12.790118+02:00 SERVERNAME kernel: e87c0060 3c820000
e8848b48 38a0fffb 4bfe6905 60000000 4bffffb4 3c620000
2021-10-10T00:13:12.790120+02:00 SERVERNAME kernel: e8638b38 7fe4fb78
480e3f55 e8410018 <0fe00000> 4bffff98 60000000 3c4c0012
2021-10-10T00:13:12.790122+02:00 SERVERNAME kernel: ---[ end trace
c9aa23777165dfdc ]---
2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error
(device dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
callbacks suppressed
2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device
dm-22): forced readonly
2021-10-10T00:13:12.790060+02:00 SERVERNAME kernel: CPU: 8 PID: 8532
Comm: vpdupdate Tainted: G W OE 4.12.14-122.83-default #1 SLE12-SP5
2021-10-10T00:13:12.790076+02:00 SERVERNAME kernel: task:
c0000068d70c1600 task.stack: c000002aba1a0000
2021-10-10T00:13:12.790078+02:00 SERVERNAME kernel: NIP:
d000000035876474 LR: d000000035876470 CTR: 0000000000000000
2021-10-10T00:13:12.790080+02:00 SERVERNAME kernel: REGS:
c000002aba1a3870 TRAP: 0700 Tainted: G W OE (4.12.14-122.83-default)
2021-10-10T00:13:12.790081+02:00 SERVERNAME kernel: MSR:
800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>
2021-10-10T00:13:12.790083+02:00 SERVERNAME kernel: CR: 24444422 XER: 20040000
2021-10-10T00:13:12.790085+02:00 SERVERNAME kernel: CFAR:
c0000000009c817c SOFTE: 1
2021-10-10T00:13:12.790086+02:00 SERVERNAME kernel: GPR00:
d000000035876470 c000002aba1a3af0 d000000035993288 0000000000000026
2021-10-10T00:13:12.790088+02:00 SERVERNAME kernel: GPR04:
c00000e6bf30ade8 c00000e6bf322a00 0000000000000007 c0000000013ed474
2021-10-10T00:13:12.790090+02:00 SERVERNAME kernel: GPR08:
0000000000000000 c000000000dc16fc 000000e6be550000 000000000000134f
2021-10-10T00:13:12.790091+02:00 SERVERNAME kernel: GPR12:
0000000000004000 c00000000f6c9400 00000000007fffff 0000000000000014
2021-10-10T00:13:12.790093+02:00 SERVERNAME kernel: GPR16:
00000100231aa080 0000010022868988 aaaaaaaaaaaaaaab c000000024ebf778
2021-10-10T00:13:12.790095+02:00 SERVERNAME kernel: GPR20:
0000000000000000 0000000000000000 0000000000000000 c00000e5a1f6eb30
2021-10-10T00:13:12.790097+02:00 SERVERNAME kernel: GPR24:
c00000e5a1f6eb20 00000000000016ba c00000e5a1f6e9c0 c00000e6828cc000
2021-10-10T00:13:12.790098+02:00 SERVERNAME kernel: GPR28:
c0000014b1bc02d0 0000000000000000 c00000e6828cc000 ffffffffffffffe4
2021-10-10T00:13:12.790100+02:00 SERVERNAME kernel: NIP
[d000000035876474] btrfs_run_delayed_refs+0x2b4/0x2c0 [btrfs]
2021-10-10T00:13:12.790101+02:00 SERVERNAME kernel: LR
[d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
2021-10-10T00:13:12.790103+02:00 SERVERNAME kernel: Call Trace:
2021-10-10T00:13:12.790104+02:00 SERVERNAME kernel: [c000002aba1a3af0]
[d000000035876470] btrfs_run_delayed_refs+0x2b0/0x2c0 [btrfs]
(unreliable)
2021-10-10T00:13:12.790106+02:00 SERVERNAME kernel: [c000002aba1a3bb0]
[d000000035891554] btrfs_commit_transaction+0x74/0xc10 [btrfs]
2021-10-10T00:13:12.790108+02:00 SERVERNAME kernel: [c000002aba1a3c80]
[d0000000358b3328] btrfs_sync_file+0x3a8/0x510 [btrfs]
2021-10-10T00:13:12.790110+02:00 SERVERNAME kernel: [c000002aba1a3d80]
[c000000000408720] vfs_fsync_range+0x70/0x120
2021-10-10T00:13:12.790111+02:00 SERVERNAME kernel: [c000002aba1a3dd0]
[c00000000040886c] do_fsync+0x5c/0xb0
2021-10-10T00:13:12.790113+02:00 SERVERNAME kernel: [c000002aba1a3e10]
[c000000000408cfc] SyS_fdatasync+0x2c/0x40
2021-10-10T00:13:12.790115+02:00 SERVERNAME kernel: [c000002aba1a3e30]
[c00000000000b308] system_call+0x3c/0x130
2021-10-10T00:13:12.790116+02:00 SERVERNAME kernel: Instruction dump:
2021-10-10T00:13:12.790118+02:00 SERVERNAME kernel: e87c0060 3c820000
e8848b48 38a0fffb 4bfe6905 60000000 4bffffb4 3c620000
2021-10-10T00:13:12.790120+02:00 SERVERNAME kernel: e8638b38 7fe4fb78
480e3f55 e8410018 <0fe00000> 4bffff98 60000000 3c4c0012
2021-10-10T00:13:12.790122+02:00 SERVERNAME kernel: ---[ end trace
c9aa23777165dfdc ]---
2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error
(device dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
callbacks suppressed
2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device

dm-22): forced readonly

$ btrfs --version
btrfs-progs v4.5.3+20160729
$ btrfs fi show /var
Label: none  uuid: f96f4980-4682-4d2d-8d7a-3c0e2c1c6680
        Total devices 1 FS bytes used 1.06GiB
        devid    1 size 2.50GiB used 2.50GiB path 
/dev/mapper/rootvg-varvol

uname -a
Linux SERVERNAME 4.12.14-122.83-default #1 SMP Tue Aug 3 08:37:22 UTC 2021 (c86c48c) ppc64le ppc64le ppc64le GNU/Linux

On the previous Friday after weekly balance:
btrfs fi usage /var
Overall:
Device size: 2.50GiB
Device allocated: 1.73GiB
Device unallocated: 792.75MiB
Device missing: 0.00B
Used: 1.09GiB
Free (estimated): 1.11GiB (min: 739.62MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00B)

Data,single: Size:1.41GiB, Used:1.08GiB
/dev/mapper/rootvg-varvol 1.41GiB

Metadata,DUP: Size:128.00MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 256.00MiB

System,DUP: Size:32.00MiB, Used:64.00KiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol     792.75MiB

I don't have extract of btrfs fi usage /var command during the weekend, but a script is extracting the Space allocated ("Size") and Used in Data and Metadata. I observed twice during the weekend space allocated to metadata is suddenly growing while the metadata used remains the same. The first time I had enough "Device unallocated" and no problem was observed, the second (on Sunday after midnight), it leads to FS RO (no space left).

Quoted 2 lines

Is there any situation that can lead to metadata allocation but without actual usage of metadata?

Same behavior this weekend (but no RO due to enough Device unallocated), something is allocating space to metadata but is not using it:

### Sat Oct 16 23:59:57 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2647.25MiB
    Device unallocated:                 2472.75MiB

Device missing: 0.00MiB
Used: 1097.75MiB
Free (estimated): 2942.38MiB (min:
1706.00MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1089.62MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:512.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1024.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 2472.75MiB

### Sun Oct 17 00:00:32 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1097.44MiB
Free (estimated): 2686.69MiB (min:
1578.31MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1089.31MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB

### Sun Oct 17 00:05:27 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1099.69MiB
Free (estimated): 2684.44MiB (min:
1576.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1091.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB

### Sun Oct 17 00:05:32 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   3159.25MiB
    Device unallocated:                 1960.75MiB

Device missing: 0.00MiB
Used: 1100.25MiB
Free (estimated): 2427.88MiB (min:
1447.50MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1092.12MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 1960.75MiB

### Sun Oct 17 00:12:53 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   3159.25MiB
    Device unallocated:                 1960.75MiB

Device missing: 0.00MiB
Used: 1100.62MiB
Free (estimated): 2427.50MiB (min:
1447.12MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1092.50MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 1960.75MiB
### Sun Oct 17 00:12:58 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1100.69MiB
Free (estimated): 2683.44MiB (min:
1575.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.12MiB)

Data,single: Size:1559.25MiB, Used:1092.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB

#2 lore raw
In reply to [email protected]
Quoted 23 lines · On Wed, Oct 13, 2021 at 02:35:39PM +0200, [email protected] wrote:

Hello,

I faced issue with btrfs FS /var forced to RO due to errno=-28 (no space left).

The server was restarted to bring back FS in RW.

Before reboot:

$ btrfs fi usage /var -m
Overall:
    Device size:         2560.00MiB
    Device allocated:         2559.00MiB
    Device unallocated:            1.00MiB
    Device missing:            0.00MiB
    Used:         1116.00MiB
    Free (estimated):          451.25MiB (min: 451.25MiB)
    Data ratio:               1.00
    Metadata ratio:               2.00
    Global reserve:           13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1108.00MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Quoted 8 lines

Metadata,DUP: Size:467.88MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 935.75MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 1.00MiB

This is a tiny filesystem, below the minimum practical size for separate data and metadata block groups, but above the size for mixed block groups to be the mkfs default. It might be better to format it with the mixed-bg option.

Quoted 3 lines

The FS went RO on Sunday, with this trace:

2021-10-10T00:13:12.790042+02:00 SERVERNAME kernel: BTRFS: Transaction

aborted (error -28)

[...]

Quoted 10 lines
2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error (device
dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
callbacks suppressed
2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device

dm-22): forced readonly

$ btrfs --version
btrfs-progs v4.5.3+20160729
Quoted 8 lines
$ btrfs fi show /var
Label: none  uuid: f96f4980-4682-4d2d-8d7a-3c0e2c1c6680
        Total devices 1 FS bytes used 1.06GiB
        devid    1 size 2.50GiB used 2.50GiB path /dev/mapper/rootvg-varvol

uname -a
Linux SERVERNAME 4.12.14-122.83-default #1 SMP Tue Aug 3 08:37:22 UTC 2021 (c86c48c) ppc64le ppc64le ppc64le GNU/Linux

This kernel is 4 years old. It may have additional practical problems in the implementation beyond the design-level issues I've described here.

Quoted 33 lines

On the previous Friday after weekly balance:
btrfs fi usage /var
Overall:
Device size: 2.50GiB
Device allocated: 1.73GiB
Device unallocated: 792.75MiB
Device missing: 0.00B
Used: 1.09GiB
Free (estimated): 1.11GiB (min: 739.62MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00B)

Data,single: Size:1.41GiB, Used:1.08GiB
/dev/mapper/rootvg-varvol 1.41GiB

Metadata,DUP: Size:128.00MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 256.00MiB

System,DUP: Size:32.00MiB, Used:64.00KiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol     792.75MiB

I don't have extract of btrfs fi usage /var command during the weekend, but a script is extracting the Space allocated ("Size") and Used in Data and Metadata. I observed twice during the weekend space allocated to metadata is suddenly growing while the metadata used remains the same. The first time I had enough "Device unallocated" and no problem was observed, the second (on Sunday after midnight), it leads to FS RO (no space left).

Quoted 2 lines

Is there any situation that can lead to metadata allocation but without actual usage of metadata?

Some btrfs maintenance operations lock block groups to prevent concurrent modification, such as balance, scrub, and discard. When this occurs, any free space that has been allocated in the metadata block groups is temporarily unavailable, and if there's no free space in any other block group, then btrfs will need to allocate another metadata block group to continue. (The same happens with data block groups but the impact is smaller due to the much larger data sizes.) The worst case for metadata on a normal filesystem is:

raid_profile_redundancy (2 for dup metadata) * (
chunk_size (usually 1GB) * (
number_of_disks (for scrub) + 1 (for discard) +
1 (for balance)
) + global_reserve (up to 512MB)
)

i.e. on a single-disk filesystem with dup metadata, there should be at least 7GB allocated or available for metadata (3.5GB dup metadata takes 7GB of raw space, you can have either the 3.5GB allocated-but-free metadata or 7GB unallocated, or any combination totalling at least 7GB).

Obviously, keeping 7GB reserved for metadata doesn't work on a device that is only 2.5GB to start with. Even if you elect not to use discard or scrub, you'd still need 2.5GB for dup metadata.

btrfs has two ways to deal with this: mixed block groups (mixed-bg mkfs option), which puts all the space into a single pool with no distinction between data and metadata, and reducing block group chunk size for non-mixed-bg filesystems.

On a filesystem this size, the block groups use 128MB chunks instead of 1GB. Global reserve is also reduced (it is computed based on device size and capped at 512MB). In this case, the worst case metadata free space requirement is 128MB chunk size * (1 disk + 2) + global_reserve which works out to 397MB of free metadata space or 794MB of unallocated space with dup metadata.

You have 467MB allocated to metadata and most of it is free space, which is theoretically enough, but if you count space in terms of block groups, it's exactly full--if the first 3 block groups are locked, the 4th block group is the last one where space might be allocated, and it's an odd size so it might not be large enough.

Mixed block groups (mixed-bg mkfs option) put all the space is in a single pot that can be allocated to metadata or data blocks as needed. This is the default for filesystems below 1GB, but it's a good idea to use mixed-bg for larger filesystems as well because the metadata block group locking cost is so high relative to the filesystem size.

While it's possible to use non-mixed block groups on a filesystem that has only a few GB, it's not possible to use a significant number of btrfs features, including scrub, balance, RAID disk replacement, online conversion to other RAID profiles, device shrink, or discard, due to the requirement to have an extra unlocked block group with available free space during these operations.

#3 lore raw
Quoted 118 lines · Zygo Blaxell
Quoted 23 lines · On Wed, Oct 13, 2021 at 02:35:39PM +0200, [email protected] wrote:

Hello,

I faced issue with btrfs FS /var forced to RO due to errno=-28 (no space left).

The server was restarted to bring back FS in RW.

Before reboot:

$ btrfs fi usage /var -m
Overall:
    Device size:         2560.00MiB
    Device allocated:         2559.00MiB
    Device unallocated:            1.00MiB
    Device missing:            0.00MiB
    Used:         1116.00MiB
    Free (estimated):          451.25MiB (min: 451.25MiB)
    Data ratio:               1.00
    Metadata ratio:               2.00
    Global reserve:           13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1108.00MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Quoted 8 lines

Metadata,DUP: Size:467.88MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 935.75MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 1.00MiB

This is a tiny filesystem, below the minimum practical size for separate data and metadata block groups, but above the size for mixed block groups to be the mkfs default. It might be better to format it with the mixed-bg option.

Quoted 3 lines

The FS went RO on Sunday, with this trace:

2021-10-10T00:13:12.790042+02:00 SERVERNAME kernel: BTRFS: Transaction

aborted (error -28)

[...]

Quoted 16 lines
2021-10-10T00:13:12.790124+02:00 SERVERNAME kernel: BTRFS: error (device
dm-22) in btrfs_run_delayed_refs:2353: errno=-28 No space left
2021-10-10T00:13:12.790125+02:00 SERVERNAME kernel: btrfs_printk: 12
callbacks suppressed
2021-10-10T00:13:12.790127+02:00 SERVERNAME kernel: BTRFS info (device

dm-22): forced readonly

$ btrfs --version
btrfs-progs v4.5.3+20160729
$ btrfs fi show /var
Label: none  uuid: f96f4980-4682-4d2d-8d7a-3c0e2c1c6680
        Total devices 1 FS bytes used 1.06GiB
        devid    1 size 2.50GiB used 2.50GiB path 
/dev/mapper/rootvg-varvol
Quoted 3 lines

uname -a
Linux SERVERNAME 4.12.14-122.83-default #1 SMP Tue Aug 3 08:37:22 UTC 2021 (c86c48c) ppc64le ppc64le ppc64le GNU/Linux

This kernel is 4 years old. It may have additional practical problems in the implementation beyond the design-level issues I've described here.

Quoted 33 lines

On the previous Friday after weekly balance:
btrfs fi usage /var
Overall:
Device size: 2.50GiB
Device allocated: 1.73GiB
Device unallocated: 792.75MiB
Device missing: 0.00B
Used: 1.09GiB
Free (estimated): 1.11GiB (min: 739.62MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00B)

Data,single: Size:1.41GiB, Used:1.08GiB
/dev/mapper/rootvg-varvol 1.41GiB

Metadata,DUP: Size:128.00MiB, Used:3.94MiB
/dev/mapper/rootvg-varvol 256.00MiB

System,DUP: Size:32.00MiB, Used:64.00KiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol     792.75MiB

I don't have extract of btrfs fi usage /var command during the weekend, but a script is extracting the Space allocated ("Size") and Used in Data and Metadata. I observed twice during the weekend space allocated to metadata is suddenly growing while the metadata used remains the same. The first time I had enough "Device unallocated" and no problem was observed, the second (on Sunday after midnight), it leads to FS RO (no space left).

Quoted 2 lines

Is there any situation that can lead to metadata allocation but without actual usage of metadata?

Some btrfs maintenance operations lock block groups to prevent concurrent modification, such as balance, scrub, and discard. When this occurs, any free space that has been allocated in the metadata block groups is temporarily unavailable, and if there's no free space in any other block group, then btrfs will need to allocate another metadata block group to continue. (The same happens with data block groups but the impact is smaller due to the much larger data sizes.) The worst case for metadata on a normal filesystem is:

raid_profile_redundancy (2 for dup metadata) * (
chunk_size (usually 1GB) * (
number_of_disks (for scrub) + 1 (for discard) +
1 (for balance)
) + global_reserve (up to 512MB)
)

Quoted 9 lines

i.e. on a single-disk filesystem with dup metadata, there should be at least 7GB allocated or available for metadata (3.5GB dup metadata takes 7GB of raw space, you can have either the 3.5GB allocated-but-free metadata or 7GB unallocated, or any combination totalling at least 7GB).

Obviously, keeping 7GB reserved for metadata doesn't work on a device that is only 2.5GB to start with. Even if you elect not to use discard or scrub, you'd still need 2.5GB for dup metadata.

Since this incident, the FS has been extended to 5GB.

But in our case the chunk size for metadata is 128MB, so:
2 [dup metadata] * ( 128 * ( 1 [disk] + 1 [discard] + 1 [balance] ) + 512 [Global Reserve] ) = 1792 MB ?

Quoted 12 lines

btrfs has two ways to deal with this: mixed block groups (mixed-bg mkfs option), which puts all the space into a single pool with no distinction between data and metadata, and reducing block group chunk size for non-mixed-bg filesystems.

On a filesystem this size, the block groups use 128MB chunks instead of 1GB. Global reserve is also reduced (it is computed based on device size and capped at 512MB). In this case, the worst case metadata free space requirement is 128MB chunk size * (1 disk + 2) + global_reserve which works out to 397MB of free metadata space or 794MB of unallocated space with dup metadata.

Quoted 6 lines

You have 467MB allocated to metadata and most of it is free space, which is theoretically enough, but if you count space in terms of block groups, it's exactly full--if the first 3 block groups are locked, the 4th block group is the last one where space might be allocated, and it's an odd size so it might not be large enough.

From obervation the last chunk allocated is not reported as totally used, so I see size for data/metdata as "odd size" (not an integer number of chunks).

Quoted 6 lines

Mixed block groups (mixed-bg mkfs option) put all the space is in a single pot that can be allocated to metadata or data blocks as needed. This is the default for filesystems below 1GB, but it's a good idea to use mixed-bg for larger filesystems as well because the metadata block group locking cost is so high relative to the filesystem size.

Quoted 6 lines

While it's possible to use non-mixed block groups on a filesystem that has only a few GB, it's not possible to use a significant number of btrfs features, including scrub, balance, RAID disk replacement, online conversion to other RAID profiles, device shrink, or discard, due to the requirement to have an extra unlocked block group with available free space during these operations.

Last weekend I had the same behavior, the size allocated to metadata went from 128MB to up to 768MB, so up to 6 * 12* MB metadata but the metdata usage didn't grow:

### Sat Oct 16 23:59:57 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2647.25MiB
    Device unallocated:                 2472.75MiB

Device missing: 0.00MiB
Used: 1097.75MiB
Free (estimated): 2942.38MiB (min:
1706.00MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1089.62MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:512.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1024.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 2472.75MiB

### Sun Oct 17 00:00:32 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1097.44MiB
Free (estimated): 2686.69MiB (min:
1578.31MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1089.31MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB

### Sun Oct 17 00:05:27 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1099.69MiB
Free (estimated): 2684.44MiB (min:
1576.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1091.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB

### Sun Oct 17 00:05:32 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   3159.25MiB
    Device unallocated:                 1960.75MiB

Device missing: 0.00MiB
Used: 1100.25MiB
Free (estimated): 2427.88MiB (min:
1447.50MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1092.12MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 1960.75MiB

### Sun Oct 17 00:12:53 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   3159.25MiB
    Device unallocated:                 1960.75MiB

Device missing: 0.00MiB
Used: 1100.62MiB
Free (estimated): 2427.50MiB (min:
1447.12MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1092.50MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 1960.75MiB
### Sun Oct 17 00:12:58 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1100.69MiB
Free (estimated): 2683.44MiB (min:
1575.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.12MiB)

Data,single: Size:1559.25MiB, Used:1092.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:
/dev/mapper/rootvg-varvol 2216.75MiB

So if I understand properly it might be due to chunk being needed for balance/discard or scrub.
Is there 3rd party tools which are also able to lock metadata block group as well? It seems to happens at the same time, when backup is running (spectrum), and/or HANA backup and/or ReaR backup as well.

#4 lore raw
Quoted 13 lines · On Tue, Oct 19, 2021 at 12:57:39PM +0200, [email protected] wrote:
Quoted 11 lines · Zygo Blaxell
Quoted 5 lines · On Wed, Oct 13, 2021 at 02:35:39PM +0200, [email protected] wrote:

Hello,

I faced issue with btrfs FS /var forced to RO due to errno=-28 (no space
left).

Obviously, keeping 7GB reserved for metadata doesn't work on a device that is only 2.5GB to start with. Even if you elect not to use discard or scrub, you'd still need 2.5GB for dup metadata.

Quoted 5 lines

Since this incident, the FS has been extended to 5GB.

But in our case the chunk size for metadata is 128MB, so:
2 [dup metadata] * ( 128 * ( 1 [disk] + 1 [discard] + 1 [balance] ) + 512 [Global Reserve] ) = 1792 MB ?

The global reserve is normally much less than 512 MB on smaller disks. It is only 13MB on the 5GB disks.

Quoted 155 lines
Quoted 6 lines

While it's possible to use non-mixed block groups on a filesystem that has only a few GB, it's not possible to use a significant number of btrfs features, including scrub, balance, RAID disk replacement, online conversion to other RAID profiles, device shrink, or discard, due to the requirement to have an extra unlocked block group with available free space during these operations.

Last weekend I had the same behavior, the size allocated to metadata went from 128MB to up to 768MB, so up to 6 * 12* MB metadata but the metdata usage didn't grow:

### Sat Oct 16 23:59:57 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2647.25MiB
    Device unallocated:                 2472.75MiB

Device missing: 0.00MiB
Used: 1097.75MiB
Free (estimated): 2942.38MiB (min: 1706.00MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1089.62MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:512.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1024.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    2472.75MiB

### Sun Oct 17 00:00:32 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1097.44MiB
Free (estimated): 2686.69MiB (min: 1578.31MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1089.31MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    2216.75MiB

### Sun Oct 17 00:05:27 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1099.69MiB
Free (estimated): 2684.44MiB (min: 1576.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1091.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    2216.75MiB

### Sun Oct 17 00:05:32 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   3159.25MiB
    Device unallocated:                 1960.75MiB

Device missing: 0.00MiB
Used: 1100.25MiB
Free (estimated): 2427.88MiB (min: 1447.50MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1092.12MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    1960.75MiB

### Sun Oct 17 00:12:53 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   3159.25MiB
    Device unallocated:                 1960.75MiB

Device missing: 0.00MiB
Used: 1100.62MiB
Free (estimated): 2427.50MiB (min: 1447.12MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.00MiB)

Data,single: Size:1559.25MiB, Used:1092.50MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    1960.75MiB
### Sun Oct 17 00:12:58 CEST 2021
Overall:
    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1100.69MiB
Free (estimated): 2683.44MiB (min: 1575.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used: 0.12MiB)

Data,single: Size:1559.25MiB, Used:1092.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Quoted 8 lines

Unallocated:

   /dev/mapper/rootvg-varvol    2216.75MiB

So if I understand properly it might be due to chunk being needed for balance/discard or scrub.
Is there 3rd party tools which are also able to lock metadata block group as well? It seems to happens at the same time, when backup is running (spectrum), and/or HANA backup and/or ReaR backup as well.

If the 3rd party tools are triggering any of the btrfs maintenance functions then they'll lock block groups; otherwise, normal filesystem operations don't generally hold locks at the block group level.

There might be some additional issues with the 4.12 kernel that cause it to allocate more than the minimum metadata. I recall there were some problems with old kernels where multiple threads allocating at the same time will all grab their own chunks, but I'm not sure which kernel those were fixed in. There are also changes to the allocator's behavior with the 'ssd' and 'nossd' mount options in 4.14 which might cause these effects.

Some temporary overallocation is normal. It's likely it would have worked even without the overallocation, the overallocation has a pretty large impact on these tiny filesystems. This seems a bit higher than the amount I've come to expect from modern btrfs.

#5 lore raw
Quoted 211 lines · Zygo Blaxell
Quoted 13 lines · On Tue, Oct 19, 2021 at 12:57:39PM +0200, [email protected] wrote:
Quoted 11 lines · Zygo Blaxell
Quoted 5 lines · On Wed, Oct 13, 2021 at 02:35:39PM +0200, [email protected] wrote:

Hello,

I faced issue with btrfs FS /var forced to RO due to errno=-28 (no space
left).

Obviously, keeping 7GB reserved for metadata doesn't work on a device that is only 2.5GB to start with. Even if you elect not to use discard or scrub, you'd still need 2.5GB for dup metadata.

Quoted 5 lines

Since this incident, the FS has been extended to 5GB.

But in our case the chunk size for metadata is 128MB, so:
2 [dup metadata] * ( 128 * ( 1 [disk] + 1 [discard] + 1 [balance] ) + 512 [Global Reserve] ) = 1792 MB ?

The global reserve is normally much less than 512 MB on smaller disks. It is only 13MB on the 5GB disks.

Quoted 167 lines
Quoted 6 lines

While it's possible to use non-mixed block groups on a filesystem that has only a few GB, it's not possible to use a significant number of btrfs features, including scrub, balance, RAID disk replacement, online conversion to other RAID profiles, device shrink, or discard, due to the requirement to have an extra unlocked block group with available free space during these operations.

Last weekend I had the same behavior, the size allocated to metadata went from 128MB to up to 768MB, so up to 6 * 12* MB metadata but the metdata usage didn't grow:

### Sat Oct 16 23:59:57 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2647.25MiB
    Device unallocated:                 2472.75MiB

Device missing: 0.00MiB
Used: 1097.75MiB
Free (estimated): 2942.38MiB (min:
1706.00MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used:
0.00MiB)

Data,single: Size:1559.25MiB, Used:1089.62MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:512.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1024.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    2472.75MiB

### Sun Oct 17 00:00:32 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1097.44MiB
Free (estimated): 2686.69MiB (min:
1578.31MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used:
0.00MiB)

Data,single: Size:1559.25MiB, Used:1089.31MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    2216.75MiB

### Sun Oct 17 00:05:27 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1099.69MiB
Free (estimated): 2684.44MiB (min:
1576.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used:
0.00MiB)

Data,single: Size:1559.25MiB, Used:1091.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    2216.75MiB

### Sun Oct 17 00:05:32 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   3159.25MiB
    Device unallocated:                 1960.75MiB

Device missing: 0.00MiB
Used: 1100.25MiB
Free (estimated): 2427.88MiB (min:
1447.50MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used:
0.00MiB)

Data,single: Size:1559.25MiB, Used:1092.12MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    1960.75MiB

### Sun Oct 17 00:12:53 CEST 2021
Overall:

    Device size:                        5120.00MiB
    Device allocated:                   3159.25MiB
    Device unallocated:                 1960.75MiB

Device missing: 0.00MiB
Used: 1100.62MiB
Free (estimated): 2427.50MiB (min:
1447.12MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used:
0.00MiB)

Data,single: Size:1559.25MiB, Used:1092.50MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:768.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1536.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Unallocated:

   /dev/mapper/rootvg-varvol    1960.75MiB
### Sun Oct 17 00:12:58 CEST 2021
Overall:
    Device size:                        5120.00MiB
    Device allocated:                   2903.25MiB
    Device unallocated:                 2216.75MiB

Device missing: 0.00MiB
Used: 1100.69MiB
Free (estimated): 2683.44MiB (min:
1575.06MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 13.00MiB (used:
0.12MiB)

Data,single: Size:1559.25MiB, Used:1092.56MiB
/dev/mapper/rootvg-varvol 1559.25MiB

Metadata,DUP: Size:640.00MiB, Used:4.00MiB
/dev/mapper/rootvg-varvol 1280.00MiB

System,DUP: Size:32.00MiB, Used:0.06MiB
/dev/mapper/rootvg-varvol 64.00MiB

Quoted 8 lines

Unallocated:

   /dev/mapper/rootvg-varvol    2216.75MiB

So if I understand properly it might be due to chunk being needed for balance/discard or scrub.
Is there 3rd party tools which are also able to lock metadata block group as well? It seems to happens at the same time, when backup is running (spectrum), and/or HANA backup and/or ReaR backup as well.

If the 3rd party tools are triggering any of the btrfs maintenance functions then they'll lock block groups; otherwise, normal filesystem operations don't generally hold locks at the block group level.

There might be some additional issues with the 4.12 kernel that cause it to allocate more than the minimum metadata. I recall there were some problems with old kernels where multiple threads allocating at the same time will all grab their own chunks, but I'm not sure which kernel those were fixed in. There are also changes to the allocator's behavior with the 'ssd' and 'nossd' mount options in 4.14 which might cause these effects.

Quoted 4 lines

Some temporary overallocation is normal. It's likely it would have worked even without the overallocation, the overallocation has a pretty large impact on these tiny filesystems. This seems a bit higher than the amount I've come to expect from modern btrfs.

Thanks for all information!

If the chunk size (128MB, 256MB... 1GB) is in relation to the FS size, is there a command to determine the chunk size for data and metadata? Should I expect BTRFS to start allocating bigger chunk at some point after filesystem extension?

#6 lore raw

On 25.10.21 г. 12:53, [email protected] wrote:
<snip>

Quoted 4 lines

Thanks for all information!

If the chunk size (128MB, 256MB... 1GB) is in relation to the FS size, is there a command to determine the chunk size for data and metadata?

You can determine it by dumping the on-disk data structure via btrfs inspect-internal dump-tree -t2 /dev/foo this will contain the currently allocated blockgroups and their respective size.

Quoted 2 lines

Should I expect BTRFS to start allocating bigger chunk at some point after filesystem extension?

Yes, this is mandated by the code in init_alloc_chunk_ctl_policy_regular function.

Basically data chunks are at most 1G in size, metadat chunks are either 1G or 256m depending on whether the fs is larger than 50gb. But a chunk can never be more than 10% of the writable space on disk.

Quoted 1 line