Skip to contentSkip to conversation

repeated enospc errors during balance on a filesystem with spare room - pls advise

DISCUSSION Ole Langbehn 5 participants 8 messages Dec 31, 2019 – Jan 1, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

I have done three full balances in a row, each of them ending with an error, telling me:

BTRFS info (device nvme1n1p1): 2 enospc errors during balance BTRFS info (device nvme1n1p1): balance: ended with status: -28

(first balance run it was 4 enospc errors).

The filesystem has enough space to spare, though:

# btrfs fi show /
Label: none  uuid: 34ea0387-af9a-43b3-b7cc-7bdf7b37b8f1
        Total devices 1 FS bytes used 624.36GiB
        devid    1 size 931.51GiB used 627.03GiB path /dev/nvme1n1p1
# btrfs fi df /
Data, single: total=614.00GiB, used=613.72GiB
System, single: total=32.00MiB, used=112.00KiB
Metadata, single: total=13.00GiB, used=10.64GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

This is after the balances, but was about the same before the balances. Before them, data had about 50GB diff between total and used.

The volume contains subvolumes (/ and /home) and snapshots (around 20 per subvolume, 40 total, oldest 1 month old).

My questions are:

1. why do I get enospc errors on a device that has enough spare space?
2. is this bad and if yes, how can I fix it?

A little more (noteworthy) context, if you're interested:

The reason I started the first balance was that a df on the filesystem showed 0% free space:

# df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/nvme1n1p1 976760584 655217424 	   0 100% /
...

and a big download (chromium sources) was aborted due to "not enough space on device".

I monitored the first balance more closely, and right after the start, df looked normal again, showing available blocks, but during the balance, it flip-flopped a couple of times between again showing 0 available bytes and showing the complement between actual size and used bytes. I did not observe this behavior any more during balance 2 and 3, but did not observe as closely.

TiA for any insights and ideas on how to proceed and a healthy start into the new year for everyone.

#1 lore raw

Excuse me for adding more information in a second mail:

# uname -a
Linux leo 5.4.6-gentoo #1 SMP PREEMPT Sun Dec 22 10:27:05 CET 2019
x86_64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz GenuineIntel GNU/Linux
# btrfs --version
btrfs-progs v5.4
#2 lore raw
Quoted 4 lines · Ole Langbehn

Excuse me for adding more information in a second mail:

# uname -a
Linux leo 5.4.6-gentoo #1 SMP PREEMPT Sun Dec 22 10:27:05 CET 2019

That is the most likely problem. A fix for 5.4's zero-free-space issue is still being reviewed on the mailing list.

On btrfs you can run out of space as often as you like, it won't hurt data on the filesystem. A lot of applications are not so robust, though, so you may encounter problems with embedded databases and configuration data. The current 5.4 bug just makes the kernel think there is 0 free space, it has nothing to do with the filesystem on disk.

Two workarounds:

- use 5.3.18 instead of 5.4.6

- use the 'metadata_ratio=1' mount option after balancing a few data block groups

In your other mail you indicated you were running a full balance. Full balances are never useful(*) and will make this specific situation worse.

A full balance includes a metadata balance. The primary effect of metadata balance is to temporarily reduce space for metadata. Reducing metadata space causes an assortment of problems for btrfs, only one of which is hitting the 5.4 free space bug. For all but a few contrived test cases, btrfs manages metadata well without interference from balance. Too much metadata balancing (i.e. any at all) can make a filesystem truly run out of metadata space on disk--a condition that is sometimes difficult to reverse.

You should run data balances to temporarily reduce space available for data. This will allow btrfs to use more space for metadata, which will avoid the bad things that happen when metadata runs out. This will also compact free space areas into large contiguous areas for future data allocations, which will improve fragmentation and may make the filesystem run a little faster.

In your specific case, you probably have to do the first few block groups of data balance on a 5.3 kernel until you have enough unallocated space to make new metadata block groups.

Going forward, run something like 'btrfs balance start -dlimit=9' every day (adjust the limit upwards if you write more than 10GB of data per day, downwards if you write less). Or use 'btrfs fi usage' and make sure that the 'unallocated' space stays above a few GB at all times, and run data balance with larger limits if it's not.

(*) A full balance contains a metadata balance, and you should never balance metadata except to convert from one RAID profile to another. Older kernels have bugs that lead to overallocation of metadata, but it is better to upgrade than to continue using the old kernels with metadata balances. There are some obscure corner test cases where a metadata balance is useful, but users will not encounter these by accident.

Quoted 5 lines

x86_64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz GenuineIntel GNU/Linux

# btrfs --version
btrfs-progs v5.4
#3 lore raw

Hi list,

Le 31/12/2019 à 23:58, Zygo Blaxell a écrit :

Quoted 8 lines

A full balance includes a metadata balance. The primary effect of metadata balance is to temporarily reduce space for metadata. Reducing metadata space causes an assortment of problems for btrfs, only one of which is hitting the 5.4 free space bug. For all but a few contrived test cases, btrfs manages metadata well without interference from balance. Too much metadata balancing (i.e. any at all) can make a filesystem truly run out of metadata space on disk--a condition that is sometimes difficult to reverse.

I however was hit by the "dummy zero free space condition" using 5.4 on a machine, and the resulting filesystem (on an external HD) then still showed 0% free using a 5.3 or a 4.15 kernel on other machines.

It however passsed "btrfs check" without any error.

The thing that fixed it and returned it to a « normal working state » has been running a "btrfs balance -m" (on the 5.4 machine) on it.

So I'm a bit puzzled reading that metadata balance is useless when it precisely fixed this issue on a FS here.

Kind regards (and best wishes for a happy new year).

#4 lore raw
Quoted 12 lines · Swâmi Petaramesh

Hi list,

Le 31/12/2019 à 23:58, Zygo Blaxell a écrit :

Quoted 8 lines

A full balance includes a metadata balance. The primary effect of metadata balance is to temporarily reduce space for metadata. Reducing metadata space causes an assortment of problems for btrfs, only one of which is hitting the 5.4 free space bug. For all but a few contrived test cases, btrfs manages metadata well without interference from balance. Too much metadata balancing (i.e. any at all) can make a filesystem truly run out of metadata space on disk--a condition that is sometimes difficult to reverse.

Quoted 3 lines

I however was hit by the "dummy zero free space condition" using 5.4 on a machine, and the resulting filesystem (on an external HD) then still showed 0% free using a 5.3 or a 4.15 kernel on other machines.

Interesting...so 5.4 can put the filesystem into a _persistent_ state where the metadata allocator makes no further progress. Two bugs in one!

I do note from the git history that metadata free space calculations seem to have been dodgy for years. The code that is touched from the 5.4 kernel bug fix patch dates back to v4.5, so probably all kernels newer than 2015 have some metadata allocation bugs (kernels up to 4.4 would hit ENOSPC while reporting non-zero free space reported in df, kernels 4.5 and later report zero free space in df first, then ENOSPC). 5.4 has other changes that make these problems more frequent, but these are not new bugs.

The longer patch series Qu mentioned looks like it's trying to properly solve the problem instead of using a quick guesstimate.

Quoted 1 line

It however passsed "btrfs check" without any error.

Not surprising. "btrfs check" cannot detect kernel bugs.

"btrfs check" is concerned with on-disk data, while free space calculations only exist in kernel memory structures. btrfs check verifies the metadata is correct, but check can't tell you if the kernel will read the metadata and then just return the wrong free space number.

Quoted 5 lines

The thing that fixed it and returned it to a « normal working state » has been running a "btrfs balance -m" (on the 5.4 machine) on it.

So I'm a bit puzzled reading that metadata balance is useless when it precisely fixed this issue on a FS here.

The kernel bug is triggered under very specific conditions. Almost any change to the filesystem could change those conditions to temporarily work around the kernel bug. Metadata balance is a change to the filesystem, so sometimes it works--but it can also trigger the problem in a filesystem that didn't have the problem before.

Other things that randomly allocate metadata, and might by chance temporarily fix this problem without the bad side-effects of metadata balances:

- metadata defrag (btrfs fi defrag on a subvol without -r)

- metadata modifications (deleting or touching files, creating and deleting empty directories)

- running a data balance (allocates a lot of metadata)

- using the metadata_ratio mount option

For 5 years our regression tests have been running on filesystems that never run metadata balances. We don't run metadata balances in the test workloads or in production because 1) in production we never have a requirement to balance metadata on any workload except for raid conversions, and 2) balancing metadata is a pretty reliable reproducer for the correct, non-bug metadata ENOSPC condition that a btrfs gets into when it runs out of both metadata and unallocated space. This is especially bad on filesystems with less than ~250GB of free space, where rounding to GB-sized block groups and free space fragmentation in data block groups combine to make it very important to have a lot of slack in metadata block groups.

We've probably been avoiding all the false-zero-free-space conditions in kernels 4.5+ accidentally, simply as a side-effect of never running a metadata balance. To reproduce the 5.4 issue, I had to set up a custom test workload which did metadata balances to force the bug to happen.

Quoted 3 lines

Kind regards (and best wishes for a happy new year).

#5 lore raw
In reply to Zygo Blaxell

Thanks for the insights (also in other responses).

Quoted 6 lines · Zygo Blaxell

Two workarounds:

- use 5.3.18 instead of 5.4.6

- use the 'metadata_ratio=1' mount option after balancing a few data block groups

Switched back to 5.3.18. Since the space bug (at least the external/df part) was solved for me before switching, I did not need to do anything further. A full(100%) data balance succeeds successfully now.

Quoted 11 lines

In your other mail you indicated you were running a full balance. Full balances are never useful(*) and will make this specific situation worse.

A full balance includes a metadata balance. The primary effect of metadata balance is to temporarily reduce space for metadata. Reducing metadata space causes an assortment of problems for btrfs, only one of which is hitting the 5.4 free space bug. For all but a few contrived test cases, btrfs manages metadata well without interference from balance. Too much metadata balancing (i.e. any at all) can make a filesystem truly run out of metadata space on disk--a condition that is sometimes difficult to reverse.

In my situation, my observation was that when I hit the space bug, I had 12GB meta reserved, 10.x used. The first balance lifted reserved space up to 13GB and I have not seen the issue afterwards. So from practice/my observations, it fixed the issue (also see Swamis response). I will take into account your insights in the future, though, and try to not balance metadata.

Cheers,

Ole

#6 lore raw
In reply to Ole Langbehn
Quoted 26 lines · Ole Langbehn

Hi,

I have done three full balances in a row, each of them ending with an error, telling me:

BTRFS info (device nvme1n1p1): 2 enospc errors during balance BTRFS info (device nvme1n1p1): balance: ended with status: -28

(first balance run it was 4 enospc errors).

The filesystem has enough space to spare, though:

# btrfs fi show /
Label: none  uuid: 34ea0387-af9a-43b3-b7cc-7bdf7b37b8f1
        Total devices 1 FS bytes used 624.36GiB
        devid    1 size 931.51GiB used 627.03GiB path /dev/nvme1n1p1
# btrfs fi df /
Data, single: total=614.00GiB, used=613.72GiB
System, single: total=32.00MiB, used=112.00KiB
Metadata, single: total=13.00GiB, used=10.64GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

This is after the balances, but was about the same before the balances. Before them, data had about 50GB diff between total and used.

Quoted 6 lines

The volume contains subvolumes (/ and /home) and snapshots (around 20 per subvolume, 40 total, oldest 1 month old).

My questions are:

1. why do I get enospc errors on a device that has enough spare space?

A known bug in v5.4, where extra space check in relocation doesn't match the new metadata over-commit.

Quoted 1 line

2. is this bad and if yes, how can I fix it?

There are patches for that:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=208445

Quoted 11 lines

A little more (noteworthy) context, if you're interested:

The reason I started the first balance was that a df on the filesystem showed 0% free space:

# df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/nvme1n1p1 976760584 655217424 	   0 100% /

Another known bug, which we also had a patch for it:
https://patchwork.kernel.org/patch/11293419/

Thanks,
Qu

Quoted 18 lines

...

and a big download (chromium sources) was aborted due to "not enough space on device".

I monitored the first balance more closely, and right after the start, df looked normal again, showing available blocks, but during the balance, it flip-flopped a couple of times between again showing 0 available bytes and showing the complement between actual size and used bytes. I did not observe this behavior any more during balance 2 and 3, but did not observe as closely.

TiA for any insights and ideas on how to proceed and a healthy start into the new year for everyone.

#7 lore raw

I learned a lot from previous replies. However, I had been in a very similar situation recently and I followed
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
so I added a pendrive to create extra space in order to be able to run btrfs balance. The extra space was not sufficient and I lost the ability to remove the pendrive. So I added another 1TB external disk, then removed the pendrive with btrfs dev remove, run the btrfs balance, then converted the profile to single/dup and then finally removed the external disk. Cost my 3 full days to overcome the issue. I'm posting this experience as a warning :)

https://unix.stackexchange.com/q/558363/65781

Qu Wenruo <[email protected]>, 1 Oca 2020 Çar, 03:13 tarihinde şunu yazdı:

Quoted 82 lines · Ole Langbehn
Quoted 26 lines

Hi,

I have done three full balances in a row, each of them ending with an error, telling me:

BTRFS info (device nvme1n1p1): 2 enospc errors during balance BTRFS info (device nvme1n1p1): balance: ended with status: -28

(first balance run it was 4 enospc errors).

The filesystem has enough space to spare, though:

# btrfs fi show /
Label: none  uuid: 34ea0387-af9a-43b3-b7cc-7bdf7b37b8f1
        Total devices 1 FS bytes used 624.36GiB
        devid    1 size 931.51GiB used 627.03GiB path /dev/nvme1n1p1
# btrfs fi df /
Data, single: total=614.00GiB, used=613.72GiB
System, single: total=32.00MiB, used=112.00KiB
Metadata, single: total=13.00GiB, used=10.64GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

This is after the balances, but was about the same before the balances. Before them, data had about 50GB diff between total and used.

Quoted 6 lines

The volume contains subvolumes (/ and /home) and snapshots (around 20 per subvolume, 40 total, oldest 1 month old).

My questions are:

1. why do I get enospc errors on a device that has enough spare space?

A known bug in v5.4, where extra space check in relocation doesn't match the new metadata over-commit.

Quoted 1 line

2. is this bad and if yes, how can I fix it?

There are patches for that:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=208445

Quoted 11 lines

A little more (noteworthy) context, if you're interested:

The reason I started the first balance was that a df on the filesystem showed 0% free space:

# df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/nvme1n1p1 976760584 655217424       0 100% /

Another known bug, which we also had a patch for it:
https://patchwork.kernel.org/patch/11293419/

Thanks,
Qu

Quoted 18 lines

...

and a big download (chromium sources) was aborted due to "not enough space on device".

I monitored the first balance more closely, and right after the start, df looked normal again, showing available blocks, but during the balance, it flip-flopped a couple of times between again showing 0 available bytes and showing the complement between actual size and used bytes. I did not observe this behavior any more during balance 2 and 3, but did not observe as closely.

TiA for any insights and ideas on how to proceed and a healthy start into the new year for everyone.