Skip to contentSkip to conversation

Endless "reclaiming chunk"/"relocating block group"

DISCUSSION Christoph Biedl 3 participants 7 messages Oct 19, 2022 – Nov 24, 2022 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hello,

On some systems I observe a strange behaviour: After remounting a BTRFS readwrite, a background process starts doing things on the disk, messages look like

| BTRFS info (device nvme0n1p1): reclaiming chunk 21486669660160 with 100% used 0% unusable
| BTRFS info (device nvme0n1p1): relocating block group 21486669660160 flags data
| BTRFS info (device nvme0n1p1): found 4317 extents, stage: move data extents
| BTRFS info (device nvme0n1p1): found 4317 extents, stage: update data pointers

and (with differing numbers) this goes on for hours and days, at a read/write rate of about 165/244 kbyte/sec. The filesystem, some 2.5 Gbyte total size, is filled to about 55%, so even if that process touches each and every block, it should already have handled everything, several times.

Now, I have no clue what is happening here, what triggers it, if it will ever finish. Point is, this takes a measuarable amount of I/O and CPU, and it delays other processes.

Some details, and things I've tested:

This behaviour is reproducible 100%, even with a btrfs created mere moments ago.

The filesystem was created using the 5.10 and 6.0 version of the btrfs-progs (both as provided by Debian stable and unstable resp.).

Using the grml rescue system (stable and daily, the latter kernel 5.19), the system does not show this behaviour.

The group block number is constantly increasing (14 digits after two days), in other words, I have not observed a wrap-around.

It was suggested in IRC to format using the --mixed parameter, no avail.

It was also suggested to set the various bg_reclaim_threshold to zero to stop this process, no avail.

This is amd64 hardware without any unusual elements. I could easily reproduce this on a fairly different platforms to make sure it's not hardware specific.

Scrubbing did not show any errors, and the problem remained.

The host runs a hand-crafted kernel, currently 5.19, and I reckon this is the source of the problem. Of course I've compared all the BTRFS kernel options, they are identical. In the block device layer configuration I couldn't see any difference that I can think would relate to this issue. Likewise I compared all kernel configuration options mentioned in src/fs/btrfs/, still nothing noteworthy.

So I'm a bit out of ideas. Unless there's something obvious from the description above, perhaps you could give a hint to the following: The process that emits the messages above, is there a way to stop it, or to report completion percentage? Looking intobtrfs_reclaim_bgs_work (block-group.c), it doesn't look like it. Are block group numbers really *that* big, magnitudes over the size of the entire filesystem?

Regards,

Christoph

#1 lore raw
Quoted 22 lines · Christoph Biedl

Hello,

On some systems I observe a strange behaviour: After remounting a BTRFS readwrite, a background process starts doing things on the disk, messages look like

| BTRFS info (device nvme0n1p1): reclaiming chunk 21486669660160 with 100% used 0% unusable
| BTRFS info (device nvme0n1p1): relocating block group 21486669660160 flags data
| BTRFS info (device nvme0n1p1): found 4317 extents, stage: move data extents
| BTRFS info (device nvme0n1p1): found 4317 extents, stage: update data pointers

and (with differing numbers) this goes on for hours and days, at a read/write rate of about 165/244 kbyte/sec. The filesystem, some 2.5 Gbyte total size, is filled to about 55%, so even if that process touches each and every block, it should already have handled everything, several times.

Now, I have no clue what is happening here, what triggers it, if it will ever finish. Point is, this takes a measuarable amount of I/O and CPU, and it delays other processes.

Quoted 7 lines

Some details, and things I've tested:

This behaviour is reproducible 100%, even with a btrfs created mere moments ago.

The filesystem was created using the 5.10 and 6.0 version of the btrfs-progs (both as provided by Debian stable and unstable resp.).

Reclaim is a purely in-kernel-memory feature, so this should not have an effect.

Quoted 5 lines

Using the grml rescue system (stable and daily, the latter kernel 5.19), the system does not show this behaviour.

The group block number is constantly increasing (14 digits after two days), in other words, I have not observed a wrap-around.

It's a 64-bit number so it's not going to wrap around any time soon.

Quoted 4 lines

It was suggested in IRC to format using the --mixed parameter, no avail.

It was also suggested to set the various bg_reclaim_threshold to zero to stop this process, no avail.

Not sure what's happening there. The reclaim threshold is the minimum amount of free space, so it shouldn't be triggering with a 100% filled block group. Reclaiming an _exactly_ filled block group makes no sense at all (no improvement of space usage is possible when a block group is completely filled), so we shouldn't be doing it.

Note that since 5.19 there are multiple bg_reclaim_threshold knobs:

/sys/fs/btrfs/(uuid)/bg_reclaim_threshold /sys/fs/btrfs/(uuid)/allocation/metadata/bg_reclaim_threshold /sys/fs/btrfs/(uuid)/allocation/system/bg_reclaim_threshold /sys/fs/btrfs/(uuid)/allocation/data/bg_reclaim_threshold

Make sure all of these are zero.

Quoted 3 lines

This is amd64 hardware without any unusual elements. I could easily reproduce this on a fairly different platforms to make sure it's not hardware specific.

It makes me think of possible rounding errors (e.g. the threshold calculation divides by 100, or there's a sum of quantities that leads to a percentage > 100, but the code treats zero as a special case and bails out long before, so I don't see how we'd reach those corner cases.

Quoted 1 line

Scrubbing did not show any errors, and the problem remained.

Scrub shouldn't interact with reclaim, except to slightly delay it.

Quoted 11 lines

The host runs a hand-crafted kernel, currently 5.19, and I reckon this is the source of the problem. Of course I've compared all the BTRFS kernel options, they are identical. In the block device layer configuration I couldn't see any difference that I can think would relate to this issue. Likewise I compared all kernel configuration options mentioned in src/fs/btrfs/, still nothing noteworthy.

So I'm a bit out of ideas. Unless there's something obvious from the description above, perhaps you could give a hint to the following: The process that emits the messages above, is there a way to stop it,

Set bg_reclaim_threshold to 0, as you've already tried (but maybe not in all of the relevant places).

Quoted 1 line

or to report completion percentage?

It's done more or less one block group at a time, so it stops when it stops. Even if it reaches the end of the block group reclaim list for one iteration of the worker, there's nothing stopping new block groups from being added while processing the list, making the percentage at any given time meaningless.

Quoted 2 lines

Are block group numbers really
*that* big, magnitudes over the size of the entire filesystem?

Relocation (including reclaim, resize, and balance) copies the data from the old block group into a new block group, removing free space fragments between the data extents in the process. Block group numbers are (almost) never reused, so each new block group created has higher bytenrs than any before it. If you've relocated every block group 100 times, the block group numbers will be over 100x the size of the filesystem.

Quoted 4 lines

Regards,

Christoph

#6 lore raw
In reply to Christoph Biedl
Quoted 11 lines · Christoph Biedl

Hello,

On some systems I observe a strange behaviour: After remounting a BTRFS readwrite, a background process starts doing things on the disk, messages look like

| BTRFS info (device nvme0n1p1): reclaiming chunk 21486669660160 with 100% used 0% unusable
| BTRFS info (device nvme0n1p1): relocating block group 21486669660160 flags data
| BTRFS info (device nvme0n1p1): found 4317 extents, stage: move data extents
| BTRFS info (device nvme0n1p1): found 4317 extents, stage: update data pointers

So that means you have automic block reclaim enabled (in this case a non-zero value in the sysfs
file /sys/fs/btrfs/<uuid>/allocation/data/bg_reclaim_threshold).

This sounds like it can be fixed by a very recent patch which is not yet in any released kernel:

https://lore.kernel.org/linux-btrfs/5f8c37f6ebc9024ef4351ae895f3e5fdb9c67baf.1665701210.git.boris@bur.io/

Quoted 57 lines

and (with differing numbers) this goes on for hours and days, at a read/write rate of about 165/244 kbyte/sec. The filesystem, some 2.5 Gbyte total size, is filled to about 55%, so even if that process touches each and every block, it should already have handled everything, several times.

Now, I have no clue what is happening here, what triggers it, if it will ever finish. Point is, this takes a measuarable amount of I/O and CPU, and it delays other processes.

Some details, and things I've tested:

This behaviour is reproducible 100%, even with a btrfs created mere moments ago.

The filesystem was created using the 5.10 and 6.0 version of the btrfs-progs (both as provided by Debian stable and unstable resp.).

Using the grml rescue system (stable and daily, the latter kernel 5.19), the system does not show this behaviour.

The group block number is constantly increasing (14 digits after two days), in other words, I have not observed a wrap-around.

It was suggested in IRC to format using the --mixed parameter, no avail.

It was also suggested to set the various bg_reclaim_threshold to zero to stop this process, no avail.

This is amd64 hardware without any unusual elements. I could easily reproduce this on a fairly different platforms to make sure it's not hardware specific.

Scrubbing did not show any errors, and the problem remained.

The host runs a hand-crafted kernel, currently 5.19, and I reckon this is the source of the problem. Of course I've compared all the BTRFS kernel options, they are identical. In the block device layer configuration I couldn't see any difference that I can think would relate to this issue. Likewise I compared all kernel configuration options mentioned in src/fs/btrfs/, still nothing noteworthy.

So I'm a bit out of ideas. Unless there's something obvious from the description above, perhaps you could give a hint to the following: The process that emits the messages above, is there a way to stop it, or to report completion percentage? Looking intobtrfs_reclaim_bgs_work (block-group.c), it doesn't look like it. Are block group numbers really *that* big, magnitudes over the size of the entire filesystem?

Regards,

Christoph

#2 lore raw
In reply to Zygo Blaxell

Thanks for add the toughts shared ...

Zygo Blaxell wrote...

(...)

Quoted 4 lines

It makes me think of possible rounding errors (e.g. the threshold calculation divides by 100, or there's a sum of quantities that leads to a percentage > 100, but the code treats zero as a special case and bails out long before, so I don't see how we'd reach those corner cases.

Out of desperation, I followed an approach I could read between the lines here. The thing I did not mention (just because I forgot about it) is the compiler version used to build the kernel. For reasons, this is still a gcc-5 (5.4.0, to be precise). After a rebuild using gcc-11, things are nice and smooth.

Now I cannot deny I'm quite confused. Assuming this is really the cause, how should I go from here? I can patch the sources at will, but it's a huge amount of code, and I don't even know where is start. Also from expericence, debug print statements do bar the optimizer from creating the problematic instructions.

Christoph

FWIW, according to Documentation/process/changes.rst, that old gcc-5 is considered sufficient. I didn't expect that :)

#3 lore raw
Quoted 10 lines · Christoph Biedl

Thanks for add the toughts shared ...

Zygo Blaxell wrote...

(...)

Quoted 4 lines

It makes me think of possible rounding errors (e.g. the threshold calculation divides by 100, or there's a sum of quantities that leads to a percentage > 100, but the code treats zero as a special case and bails out long before, so I don't see how we'd reach those corner cases.

Quoted 5 lines

Out of desperation, I followed an approach I could read between the lines here. The thing I did not mention (just because I forgot about it) is the compiler version used to build the kernel. For reasons, this is still a gcc-5 (5.4.0, to be precise). After a rebuild using gcc-11, things are nice and smooth.

That's...not the weirdest thing I've ever seen, but maybe the weirdest thing I've seen this month.

Quoted 5 lines

Now I cannot deny I'm quite confused. Assuming this is really the cause, how should I go from here? I can patch the sources at will, but it's a huge amount of code, and I don't even know where is start. Also from expericence, debug print statements do bar the optimizer from creating the problematic instructions.

That in itself would (weakly) confirm a compiler issue. But it might be simpler than that, like a bug in the implementation of a math builtin that wasn't popular in the kernel 8 years ago, but is popular now after the implementation was debugged in GCC. I don't know if that's the case in this instance, but similar things have happened in the past.

Quoted 4 lines

Christoph

FWIW, according to Documentation/process/changes.rst, that old gcc-5 is considered sufficient. I didn't expect that :)

On paper maybe, but maybe not in real-world use. Somewhat related thread on linux-kernel:

	https://lore.kernel.org/all/[email protected]/T/

TL;DR not enough people are testing new kernel code against old compilers. If it's problematic to keep the host system's gcc up to date, a build chroot for kernel building with an up to date toolchain is the way to go.

#4 lore raw

Zygo Blaxell wrote...

Quoted 3 lines

TL;DR not enough people are testing new kernel code against old compilers. If it's problematic to keep the host system's gcc up to date, a build chroot for kernel building with an up to date toolchain is the way to go.

Indeed, but unfortunately this is not an option in the given environment.

Eventually I started bisecting and found the commit below introduced the trouble. And just to make it clear, the change is correct as far as I can see. The code built by gcc 5.4 is not.

commit ac2f1e63c65c695b6134f40a078cf82df627e188

Author: Josef Bacik <[email protected]>
Date:   Tue Mar 29 01:56:07 2022 -0700

btrfs: allow block group background reclaim for non-zoned filesystems

These final three hunks caught my attention:

@@ -3220,6 +3245,8 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,

        spin_unlock(&info->delalloc_root_lock);

        while (total) {
+               bool reclaim;
+
                cache = btrfs_lookup_block_group(info, bytenr);
                if (!cache) {
                        ret = -ENOENT;
@@ -3265,6 +3292,8 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
                                        cache->space_info, num_bytes);
                        cache->space_info->bytes_used -= num_bytes;
                        cache->space_info->disk_used -= num_bytes * factor;
+
+                       reclaim = should_reclaim_block_group(cache, num_bytes);
                        spin_unlock(&cache->lock);
                        spin_unlock(&cache->space_info->lock);

@@ -3291,6 +3320,8 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,

                if (!alloc && old_val == 0) {
                        if (!btrfs_test_opt(info, DISCARD_ASYNC))
                                btrfs_mark_bg_unused(cache);
+               } else if (!alloc && reclaim) {
+                       btrfs_mark_bg_to_reclaim(cache);
                }

                btrfs_put_block_group(cache);

It seems strange "reclaim" is not initialized but after a closer look this turns out to be okay. That variable is initialized if "alloc" is false (not visible in the diff above), but only used under that condition as well (last hunk). Which is why gcc does not emit warnings about this.

*However*, some additional debug-print statements revealed the generated code enters the block that calls btrfs_mark_bg_to_reclaim /even/ /if/ alloc is false, and reclaim undefined (usually true).

That is fairy scary, and I have no idea how many other places in the kernel are affected by this. I'd assume, quite a few.

Workaround is trivial by initializing reclaim to false. But still.

Christoph

#5 lore raw

Stupid typo ...

Christoph Biedl wrote...

Quoted 3 lines

*However*, some additional debug-print statements revealed the generated code enters the block that calls btrfs_mark_bg_to_reclaim /even/ /if/ alloc is

true,

Quoted 1 line

and reclaim undefined (usually true).