Skip to contentSkip to conversation

generic/301: flaky failure on btrfs after metadata overcommit change

RFC Leo Martins 3 participants 3 messages Mar 23 – Mar 24 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

generic/301 has become flaky on btrfs after commit 0dc118b3c327 ("btrfs:
be less aggressive with metadata overcommit when we can do full flushing") which landed in btrfs/for-next. Out of 30 runs, 8 fail with:

+file2 badly fragmented

I bisected this to the above commit, which reduces the metadata overcommit limit from 1/8th to 1/64th of available space for full-flushing contexts. This is a legitimate fix for -ENOSPC transaction aborts on small filesystems, but as a side effect it causes more frequent transaction commits during writeback. The reduced batching means the extent allocator has less opportunity to coalesce adjacent CoW extents, resulting in higher extent counts that sometimes cross the test's threshold.

The fragmentation check in question is:

    test $new_extents -lt $((internal_blks * 2 / 3)) || echo "file2 badly fragmented"

The 2/3 threshold was introduced in 9184ca155d7c ("xfs: test fragmentation characteristics of copy-on-write") as part of a series testing XFS's CoW extent size hint (cowextsize) mechanism. For btrfs, this threshold is arbitrary — btrfs doesn't have XFS's cowextsize hint, and its CoW extent allocation depends on factors like transaction commit frequency and metadata reservation behavior, which is exactly what the overcommit commit changed.

I see two possible fixes and would appreciate input on which is preferred:

Option A: _notrun for btrfs
----------------------------

Skip the entire test since the fragmentation threshold is not applicable to btrfs:

test $FSTYP = "btrfs" && \
_notrun "CoW fragmentation threshold not applicable to btrfs"

Option B: Skip only the extent count assertion for btrfs
---------------------------------------------------------

Keep the CoW + data integrity portion of the test (the md5sum checks after random CoW writes and remount are still useful) and only skip the fragmentation assertion:

    if [ "$FSTYP" != "btrfs" ]; then
        test $new_extents -lt $((internal_blks * 2 / 3)) || \
            echo "file2 badly fragmented"
    fi

I lean towards option B since the CoW write + remount + md5sum verification is still a reasonable smoke test, but option A is cleaner if the consensus is that this test isn't adding value for btrfs.

Thoughts?

Thanks,
Leo Martins

#1 lore raw
Quoted 53 lines · Leo Martins

Hi,

generic/301 has become flaky on btrfs after commit 0dc118b3c327 ("btrfs:
be less aggressive with metadata overcommit when we can do full flushing") which landed in btrfs/for-next. Out of 30 runs, 8 fail with:

+file2 badly fragmented

I bisected this to the above commit, which reduces the metadata overcommit limit from 1/8th to 1/64th of available space for full-flushing contexts. This is a legitimate fix for -ENOSPC transaction aborts on small filesystems, but as a side effect it causes more frequent transaction commits during writeback. The reduced batching means the extent allocator has less opportunity to coalesce adjacent CoW extents, resulting in higher extent counts that sometimes cross the test's threshold.

The fragmentation check in question is:

    test $new_extents -lt $((internal_blks * 2 / 3)) || echo "file2 badly fragmented"

The 2/3 threshold was introduced in 9184ca155d7c ("xfs: test fragmentation characteristics of copy-on-write") as part of a series testing XFS's CoW extent size hint (cowextsize) mechanism. For btrfs, this threshold is arbitrary — btrfs doesn't have XFS's cowextsize hint, and its CoW extent allocation depends on factors like transaction commit frequency and metadata reservation behavior, which is exactly what the overcommit commit changed.

I see two possible fixes and would appreciate input on which is preferred:

Option A: _notrun for btrfs
----------------------------

Skip the entire test since the fragmentation threshold is not applicable to btrfs:

test $FSTYP = "btrfs" && \
_notrun "CoW fragmentation threshold not applicable to btrfs"

Option B: Skip only the extent count assertion for btrfs
---------------------------------------------------------

Keep the CoW + data integrity portion of the test (the md5sum checks after random CoW writes and remount are still useful) and only skip the fragmentation assertion:

    if [ "$FSTYP" != "btrfs" ]; then
        test $new_extents -lt $((internal_blks * 2 / 3)) || \
            echo "file2 badly fragmented"
    fi
Quoted 5 lines

I lean towards option B since the CoW write + remount + md5sum verification is still a reasonable smoke test, but option A is cleaner if the consensus is that this test isn't adding value for btrfs.

Thoughts?

B, since it's checking data integrity across a mount cycle.

--D

Quoted 3 lines

Thanks,
Leo Martins

#2 lore raw
In reply to Leo Martins
Quoted 48 lines · Leo Martins

Hi,

generic/301 has become flaky on btrfs after commit 0dc118b3c327 ("btrfs:
be less aggressive with metadata overcommit when we can do full flushing") which landed in btrfs/for-next. Out of 30 runs, 8 fail with:

+file2 badly fragmented

I bisected this to the above commit, which reduces the metadata overcommit limit from 1/8th to 1/64th of available space for full-flushing contexts. This is a legitimate fix for -ENOSPC transaction aborts on small filesystems, but as a side effect it causes more frequent transaction commits during writeback. The reduced batching means the extent allocator has less opportunity to coalesce adjacent CoW extents, resulting in higher extent counts that sometimes cross the test's threshold.

The fragmentation check in question is:

     test $new_extents -lt $((internal_blks * 2 / 3)) || echo "file2 badly fragmented"

The 2/3 threshold was introduced in 9184ca155d7c ("xfs: test fragmentation characteristics of copy-on-write") as part of a series testing XFS's CoW extent size hint (cowextsize) mechanism. For btrfs, this threshold is arbitrary — btrfs doesn't have XFS's cowextsize hint, and its CoW extent allocation depends on factors like transaction commit frequency and metadata reservation behavior, which is exactly what the overcommit commit changed.

I see two possible fixes and would appreciate input on which is preferred:

Option A: _notrun for btrfs
----------------------------

Skip the entire test since the fragmentation threshold is not applicable to btrfs:

test $FSTYP = "btrfs" && \
_notrun "CoW fragmentation threshold not applicable to btrfs"

Option B: Skip only the extent count assertion for btrfs
---------------------------------------------------------

Keep the CoW + data integrity portion of the test (the md5sum checks after random CoW writes and remount are still useful) and only skip the fragmentation assertion:

Quoted 8 lines
     if [ "$FSTYP" != "btrfs" ]; then
         test $new_extents -lt $((internal_blks * 2 / 3)) || \
             echo "file2 badly fragmented"
     fi

I lean towards option B since the CoW write + remount + md5sum verification is still a reasonable smoke test, but option A is cleaner if the consensus is that this test isn't adding value for btrfs.

I also agree on the option B.

The fragmentation behavior is really specific to each fs, thus leaving the number of extents test to xfs, and keeping the contents check looks good to me.

Thanks,
Qu

Quoted 6 lines

Thoughts?

Thanks,
Leo Martins