ERROR: scrubbing <dev> failed for device id 1: ret=-1, errno=28 (No space left on device)

Hello,

checking all my btr file systems after seeing a lot of trouble on a particular installation, I came across this:

# btrfs scrub start -B -c3 /mnt/schroot/
ERROR: scrubbing /mnt/schroot/ failed for device id 1: ret=-1, errno=28 (No space left on device)
scrub canceled for 89a02165-5975-46c7-8565-1247874531a2
        scrub started at Fri Dec 18 08:20:27 2015 and was aborted after 00:00:01
        total bytes scrubbed: 7.70MiB with 0 errors

btrfsck shows a *lot* of the "bad extent [458797056, 458801152), type mismatch with chunk" messages that have been discussed here but otherwise exits zero.

df shows 75% usage (4.4 of 6.5 Gibyte used)).

# btrfs fi df /mnt/schroot/
Data, single: total=5.17GiB, used=3.75GiB
System, DUP: total=8.00MiB, used=4.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=409.56MiB, used=143.44MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=48.00MiB, used=0.00B

This is btrfs-progs v4.3 running on kernel 4.3.3

Any idea how to proceed?

Christoph
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

you need to do some balancing, as I think that the free space is too fragmented and the system fails to allocate extra metadata space needed for scrub doing writes into metadata.

# btrfs balance start -dusage=<number>
with number somewhere between 5 and 50; first start with lower number
and repeat and increase until the 5.17GiB gets closer down to the
3.75GiB

The issue is that for for a filesystem of this size (6.5GiB), it would have been better to have mixed data/metadata allocation. See help of mkfs.btrfs. So maybe you want to recreate the fs like that, otherwise you might easily hit this problem again if you don't monitor free space / balance regularly.

Henk Slager wrote...

Quoted 8 lines

you need to do some balancing, as I think that the free space is too fragmented and the system fails to allocate extra metadata space needed for scrub doing writes into metadata.

# btrfs balance start -dusage=<number>
with number somewhere between 5 and 50; first start with lower number
and repeat and increase until the 5.17GiB gets closer down to the
3.75GiB

Thanks, that one worked with number 100. Still some ENOSPC errors but appeareantly good enough to make scrubbing work.

Quoted 5 lines

The issue is that for for a filesystem of this size (6.5GiB), it would have been better to have mixed data/metadata allocation. See help of mkfs.btrfs. So maybe you want to recreate the fs like that, otherwise you might easily hit this problem again if you don't monitor free space / balance regularly.

Will keep that in mind. Just out of curiousity: Initially, the file system size was just 4 Gibyte. Is it possible this made things worse?

Oh, and by the way:

Documentation/mkfs.btrfs.asciidoc +1 -1

@@ -77,7 +77,7 @@ where the space is reserved for the other block group type, is not available for
 allocation and can lead to ENOSPC state.
 +
 The recommended size for the mixed mode is for filesystems less than 1GiB. The
-soft recommendation is to use it for filesystems smaller than 5GiB. Thie mixed
+soft recommendation is to use it for filesystems smaller than 5GiB. The mixed
 mode may lead to degraded performance on larger filesystems, but is otherwise
 usable, even on multiple devices.
 +

Christoph
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html