Skip to contentSkip to conversation

strange "No space left on device"

DISCUSSION Christoph Anton Mitterer 5 participants 8 messages Nov 7, 2015 – Nov 9, 2015 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hey.

I just repeatedly did the following twice on a ~8GB USB stick, under Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).

First, created some GPT on the stick:
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048575   511.0 MiB   EF02  BIOS boot partition 2         1048576         3145727   1024.0 MiB  8300  Linux filesystem 3         3145728         4194303   512.0 MiB   8300  Linux filesystem 4         4194304         6291455   1024.0 MiB  8300  Linux filesystem 5         6291456        15687644   4.5 GiB     8300  Linux filesystem

Then filesystems:

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2 
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.

Label:              boot-main
UUID:               150ee9fb-c650-4b5b-8f64-606e589e429a Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP              59.19MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb2

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.

Label:              boot-data
UUID:               b1c1fc77-c965-4f0c-a2b5-44a301fd8772 Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP              59.19MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb3

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive
/dev/sdb4
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.

Label:              boot-archive
UUID:               a063cf3b-0322-4ec7-a8c1-2dabecad9f57 Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP              59.19MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb4

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.

Label:              boot-rescue
UUID:               104b7bc3-3b8c-4a08-b0a6-9172ed664e68 Node size:          16384
Sector size:        4096
Filesystem size:    3.98GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP             211.75MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     3.98GiB  /dev/sdb5

No errors in the kernel log.

Then I got usage info:

root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/
Overall:
    Device size:		   1.00GiB
    Device allocated:		 126.38MiB
    Device unallocated:		 897.62MiB
    Device missing:		     0.00B
    Used:			 256.00KiB
    Free (estimated):		 905.62MiB	(min: 456.81MiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Data,single: Size:8.00MiB, Used:0.00B
   /dev/sdb3    8.00MiB

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
   /dev/sdb3  102.38MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdb3   16.00MiB

Unallocated:
   /dev/sdb3  897.62MiB

root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/
Overall:
    Device size:		   1.00GiB
    Device allocated:		 126.38MiB
    Device unallocated:		 897.62MiB
    Device missing:		     0.00B
    Used:			 256.00KiB
    Free (estimated):		 905.62MiB	(min: 456.81MiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Data,single: Size:8.00MiB, Used:0.00B
   /dev/sdb2    8.00MiB

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
   /dev/sdb2  102.38MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdb2   16.00MiB

Unallocated:
   /dev/sdb2  897.62MiB

root@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/
Overall:
    Device size:		   3.98GiB
    Device allocated:		 431.50MiB
    Device unallocated:		   3.56GiB
    Device missing:		     0.00B
    Used:			 320.00KiB
    Free (estimated):		   3.57GiB	(min: 1.79GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Data,single: Size:8.00MiB, Used:64.00KiB
   /dev/sdb5    8.00MiB

Metadata,DUP: Size:203.75MiB, Used:112.00KiB
   /dev/sdb5  407.50MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdb5   16.00MiB

Unallocated:
   /dev/sdb5    3.56GiB

Still all fine.

Now I write to one of the filesystems (main at first, that still works) but then when I cp -a from another btrfs to data:
cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
cp: error writing ‘data/SHA512-sums’: No space left on device
cp: failed to extend ‘data/SHA512-sums’: No space left on device

(these files are just a few bytes large)
On the target fs, they're there but all 0 bytes large.

(btw: I've aliased cp to cp --reflink=auto

When I now repeat the usage info:

root@heisenberg:/data/SSS/boot# btrfs filesystem usage data
Overall:
    Device size:		   1.00GiB
    Device allocated:		 118.38MiB
    Device unallocated:		 905.62MiB
    Device missing:		     0.00B
    Used:			 256.00KiB
    Free (estimated):		   8.00EiB	(min: 8.00EiB)
    Data ratio:			      -nan
    Metadata ratio:		      2.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
   /dev/sdc3  102.38MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdc3   16.00MiB

Unallocated:
   /dev/sdc3  905.62MiB

It get's bogus... at least the Free space shows 8 EiB (if wishes were horses)...

Still no error in kernel logs:
 3401.934514] BTRFS: device label boot-rescue devid 1 transid 8 /dev/sdc5

[ 3401.985858] BTRFS: device label boot-main devid 1 transid 8
/dev/sdc2
[ 3401.988601] BTRFS: device label boot-data devid 1 transid 9
/dev/sdc3
[ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9
/dev/sdc4
[ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled
[ 3403.721098] BTRFS: has skinny extents
[ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled
[ 3413.964042] BTRFS: has skinny extents
[ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled
[ 3522.902314] BTRFS: has skinny extents
[ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled
[ 3525.055659] BTRFS: has skinny extents

btrfs check doesn't show errors either.

After a balance, i can copy again, even though the usage still shows 8 EiB for a while...

Anyway how can it happen, that on a fresh btrfs with no files at all a balance is necessary?
Or is there some deeper corruption going on here?

Thanks,
Chris.

#1 lore raw
Quoted 5 lines · Christoph Anton Mitterer

Hey.

I just repeatedly did the following twice on a ~8GB USB stick, under Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).

Quoted 7 lines

First, created some GPT on the stick:
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048575   511.0 MiB   EF02  BIOS boot partition 2         1048576         3145727   1024.0 MiB  8300  Linux filesystem 3         3145728         4194303   512.0 MiB   8300  Linux filesystem 4         4194304         6291455   1024.0 MiB  8300  Linux filesystem 5         6291456        15687644   4.5 GiB     8300  Linux filesystem

These are all really small.

I would suggest running mkfs with --mixed for all of these
filesystems and trying again.

Hugo.

Quoted 234 lines

Then filesystems:

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2 
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.

Label:              boot-main
UUID:               150ee9fb-c650-4b5b-8f64-606e589e429a Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP              59.19MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb2

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.

Label:              boot-data
UUID:               b1c1fc77-c965-4f0c-a2b5-44a301fd8772 Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP              59.19MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb3

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive
/dev/sdb4
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.

Label:              boot-archive
UUID:               a063cf3b-0322-4ec7-a8c1-2dabecad9f57 Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP              59.19MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb4

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.

Label:              boot-rescue
UUID:               104b7bc3-3b8c-4a08-b0a6-9172ed664e68 Node size:          16384
Sector size:        4096
Filesystem size:    3.98GiB
Block group profiles:
  Data:             single            8.00MiB Metadata:         DUP             211.75MiB System:           DUP              12.00MiB SSD detected:       no
Incompat features:  extref, skinny-metadata Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     3.98GiB  /dev/sdb5

No errors in the kernel log.

Then I got usage info:

root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/
Overall:
    Device size:		   1.00GiB
    Device allocated:		 126.38MiB
    Device unallocated:		 897.62MiB
    Device missing:		     0.00B
    Used:			 256.00KiB
    Free (estimated):		 905.62MiB	(min: 456.81MiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Data,single: Size:8.00MiB, Used:0.00B
   /dev/sdb3    8.00MiB

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
   /dev/sdb3  102.38MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdb3   16.00MiB

Unallocated:
   /dev/sdb3  897.62MiB

root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/
Overall:
    Device size:		   1.00GiB
    Device allocated:		 126.38MiB
    Device unallocated:		 897.62MiB
    Device missing:		     0.00B
    Used:			 256.00KiB
    Free (estimated):		 905.62MiB	(min: 456.81MiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Data,single: Size:8.00MiB, Used:0.00B
   /dev/sdb2    8.00MiB

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
   /dev/sdb2  102.38MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdb2   16.00MiB

Unallocated:
   /dev/sdb2  897.62MiB

root@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/
Overall:
    Device size:		   3.98GiB
    Device allocated:		 431.50MiB
    Device unallocated:		   3.56GiB
    Device missing:		     0.00B
    Used:			 320.00KiB
    Free (estimated):		   3.57GiB	(min: 1.79GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Data,single: Size:8.00MiB, Used:64.00KiB
   /dev/sdb5    8.00MiB

Metadata,DUP: Size:203.75MiB, Used:112.00KiB
   /dev/sdb5  407.50MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdb5   16.00MiB

Unallocated:
   /dev/sdb5    3.56GiB

Still all fine.

Now I write to one of the filesystems (main at first, that still works) but then when I cp -a from another btrfs to data:
cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
cp: error writing ‘data/SHA512-sums’: No space left on device
cp: failed to extend ‘data/SHA512-sums’: No space left on device

(these files are just a few bytes large)
On the target fs, they're there but all 0 bytes large.

(btw: I've aliased cp to cp --reflink=auto

When I now repeat the usage info:

root@heisenberg:/data/SSS/boot# btrfs filesystem usage data
Overall:
    Device size:		   1.00GiB
    Device allocated:		 118.38MiB
    Device unallocated:		 905.62MiB
    Device missing:		     0.00B
    Used:			 256.00KiB
    Free (estimated):		   8.00EiB	(min: 8.00EiB)
    Data ratio:			      -nan
    Metadata ratio:		      2.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
   /dev/sdc3  102.38MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sdc3   16.00MiB

Unallocated:
   /dev/sdc3  905.62MiB

It get's bogus... at least the Free space shows 8 EiB (if wishes were horses)...

Still no error in kernel logs:
 3401.934514] BTRFS: device label boot-rescue devid 1 transid 8 /dev/sdc5

[ 3401.985858] BTRFS: device label boot-main devid 1 transid 8
/dev/sdc2
[ 3401.988601] BTRFS: device label boot-data devid 1 transid 9
/dev/sdc3
[ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9
/dev/sdc4
[ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled
[ 3403.721098] BTRFS: has skinny extents
[ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled
[ 3413.964042] BTRFS: has skinny extents
[ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled
[ 3522.902314] BTRFS: has skinny extents
[ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled
[ 3525.055659] BTRFS: has skinny extents

btrfs check doesn't show errors either.

After a balance, i can copy again, even though the usage still shows 8 EiB for a while...

Anyway how can it happen, that on a fresh btrfs with no files at all a balance is necessary?
Or is there some deeper corruption going on here?

Thanks,
Chris.

Signature · 4 lines
Hugo Mills             | Reintarnation: Coming back from the dead as a
hugo@... carfax.org.uk | hillbilly
http://carfax.org.uk/  |
PGP: E2AB1DE4          |
#2 lore raw
Quoted 1 line · Hugo Mills

   These are all really small.

Well enough for booting =)

Quoted 2 lines

   I would suggest running mkfs with --mixed for all of these filesystems and trying again.

I thought btrfs would do that automatically:
https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_small
"Using mixed block groups is recommended for filesystems of 1GiB or smaller and mkfs.btrfs will force mixed block groups automatically in that case."

Anyway I even if that's the reason, than I think something's quite wrong here... a) it doesn't happen always (I just create 10 times a fs in the same partition, and no problem) ... b) in those cases where I could produce the issue, it went away after a balance... so perhaps they just just balance the fs right after mkfs.... o.O

Cheers,
Chris.

#3 lore raw
In reply to Christoph Anton Mitterer

[...]

Quoted 1 line
   3         3145728         4194303   512.0 MiB   8300  Linux filesystem

[...]

Quoted 4 lines
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Quoted 15 lines
Label:              boot-data
UUID:               b1c1fc77-c965-4f0c-a2b5-44a301fd8772
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              59.19MiB
  System:           DUP              12.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb3

Somehow your mkfs.btrfs (or some other part of the system) assumed the wrong size for partition 3. To me, that sounds like a possible cause of the errors and 'EiB' reported size.
For RAID tests in VMs I have also been using 4-8GB disk sizes (instead of TB sized like the real hardware) and I encountered many 'No space left on device' message (with older kernels).
A quick test with kernel 4.3.0 now:

# dd if=/dev/zero of=sd.bin bs=1M count=512
# losetup -f sd.bin
# mkfs.btrfs --nodiscard --label boot-data /dev/loop0
SMALL VOLUME: forcing mixed metadata/data groups
btrfs-progs v4.2.3+20151102
See http://btrfs.wiki.kernel.org for more information.
Label:              boot-data
UUID:               f152781b-b96b-4969-96b8-eede3e3dabc2
Node size:          4096
Sector size:        4096
Filesystem size:    512.00MiB
Block group profiles:
  Data+Metadata:    single            8.00MiB
  System:           single            4.00MiB
SSD detected:       no
Incompat features:  mixed-bg, extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1   512.00MiB  /dev/loop0
#4 lore raw

Henk Slager posted on Sun, 08 Nov 2015 19:18:19 +0100 as excerpted:

Quoted 22 lines

[...]

Quoted 1 line
   3     3145728     4194303   512.0 MiB   8300  Linux filesystem

[...]

Quoted 4 lines
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more
information.
Quoted 14 lines
Label:              boot-data
UUID:               b1c1fc77-c965-4f0c-a2b5-44a301fd8772
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              59.19MiB
  System:           DUP              12.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1 Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb3
Quoted 3 lines

Somehow your mkfs.btrfs (or some other part of the system) assumed the wrong size for partition 3. To me, that sounds like a possible cause of the errors and 'EiB' reported size.

Wow, yes! Good catch, Henk! =:^) Hugo obviously didn't catch it, and I wouldn't have either, as the bad size detection behavior is so unexpected, it just wouldn't occur to me to look!

Something's clearly wrong, and as it could clearly either overwrite the next partition(s) if the partitioning detection is at fault and doesn't stop it, or throw all sorts of monkey wrenches into btrfs behavior if it does, this is rather alarming behavior indeed! Clearly we need devs to take a look and figure out what bugged out, and how to fix it.

That potentially explains why you got separate data/metadata instead of mixed mode, and why it worked most of the time but failed occasionally, since presumably when it worked the size detection worked as well.

(Apparently, btrfs-progs-4.3 does away with the default to mixed-mode at 1 GiB or under, tho it is still recommended. I'm not exactly sure of why, tho I think it had to do with being able to use sub-GiB btrfs for testing without having to worry about mixed mode. Still, one would think, since it's recommended, that they'd simply provide an option to make it split mode, instead of changing the default to something that's not recommended. So from 4.3 userspace you'd have to specifically set mixed, but you're on 4.2.2 userspace, so that shouldn't be the case yet. And in any case that doesn't explain the very badly detected size and an attempt to create a 1 GiB btrfs on a 1/2 GiB partition, which is obviously the real problem, regardless of whatever layer it occurred in.)

Meanwhile, btrfs fi usage doesn't understand mixed-mode yet, and returns 8 EiB for some values if it is used, which is originally what I thought was happening. But the problem is elsewhere, as it wasn't mixed mode, but rather, a very /very/ screwed up size detection.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#5 lore raw
Quoted 4 lines · Duncan

Wow, yes!  Good catch, Henk! =:^)  Hugo obviously didn't catch it, and I
wouldn't have either, as the bad size detection behavior is so unexpected, it just wouldn't occur to me to look!

Hmm... all that *may* be more likely an error of myself when copying and pasting the terminal output together:

I did actually change the 3rd partition to use 1GiB in later tries at the expense of the 5th one shrinking, so the part table would have looked like this in these later tries:
512M
1G
1G
1G
4G

Which would again fit the output of the various mkfs.btrfs.

Sorry if that was the case, apologies for any confusion.

The problem seemed to went away when explicitly using --mixed.

Quoted 2 lines

(Apparently, btrfs-progs-4.3 does away with the default to mixed- mode at 1 GiB or under, tho it is still recommended.

Well I still had 4.2 ...

Quoted 4 lines

I'm not exactly sure of
why, tho I think it had to do with being able to use sub-GiB btrfs for
testing without having to worry about mixed mode.

Kinda strange... shouldn't it work out of the box for users and not developers?

To be honest, no one should need to read through the wiki, just to be able to create a small sized fs.

And even if no mixed D/M block group allocation was used... it shouldn't just fail out-of-the-box with a few byte large files on a 1 GB fs.

Cheers,
Chris.

#6 lore raw

Christoph Anton Mitterer posted on Sun, 08 Nov 2015 23:10:57 +0100 as excerpted:

Quoted 11 lines · Duncan
Quoted 3 lines

Wow, yes!  Good catch, Henk! =:^)  Hugo obviously didn't catch it, and I wouldn't have either, as the bad size detection behavior is so unexpected, it just wouldn't occur to me to look!

Hmm... all that *may* be more likely an error of myself when copying and pasting the terminal output together:

I did actually change the 3rd partition to use 1GiB in later tries at the expense of the 5th one shrinking, so the part table would have looked like this in these later tries:
512M 1G 1G 1G 4G

Oops! =:^(

Quoted 6 lines

Which would again fit the output of the various mkfs.btrfs.

Sorry if that was the case, apologies for any confusion.

The problem seemed to went away when explicitly using --mixed.

Good. Whatever the cause of it not doing mixed by default (perhaps it's an off-by-one error and filesystems of exactly 1 GiB weren't defaulting to mixed as expected), the fact that mixed was the default on small filesystems would mean separate data/metadata wouldn't have gotten the same full testing coverage, so maybe there's a bug... In which case the 4.3 change discussed below would certainly tend to trigger it more often, hopefully getting it fixed. =:^)

Quoted 4 lines
Quoted 2 lines

(Apparently, btrfs-progs-4.3 does away with the default to mixed- mode at 1 GiB or under, tho it is still recommended.

Well I still had 4.2 ...

That's why I had that as a parenthetical. It didn't apply to your current case, but could to others, and I've learned to write with both the lurkers and the googlers who will happen on my post later, in mind.

Quoted 9 lines
Quoted 3 lines

I'm not exactly sure of why, tho I think it had to do with being able to use sub-GiB btrfs for testing without having to worry about mixed mode.

Kinda strange... shouldn't it work out of the box for users and not developers?

To be honest, no one should need to read through the wiki, just to be able to create a small sized fs.

That's why, tho I didn't press it, I asked for clarification when I saw it mentioned on the btrfs-progs 4.3-rc1 announcement, verifying that mixed was still recommended. Chris Murphy (another non-btrfs-dev list regular, I think he works or volunteers for one of the distros) questioned it as well, on the same announcement.

But not being able to personally show-them-the-code, as long as there's at least an option to do it the way I want (which there is, the already mentioned --mixed option), I've learned not to press it.[1]

Quoted 2 lines

And even if no mixed D/M block group allocation was used... it shouldn't just fail out-of-the-box with a few byte large files on a 1 GB fs.

You are absolutely correct. There's a bug somewhere. Earlier I thought it was the size detection bug Henk spotted, since even if there's some other bug here it'd pale in comparision, but since that bug turned out to trace to your copy/paste, that isn't it, and we still have the much more ordinary level bug to deal with.

But as I said above, with progs 4.3 defaulting to unmixed even on small filesystems now, if indeed it is tied to unmixed on a small filesystem, that bug's very likely to get MUCH more common, and thus should be fixed relatively quickly, I'd guess by the time 4.4 comes out, given the history of other bugs that turn out to be quite common.

---
[1] Tho for my own systems I sometimes patch in the behavior as I think it should be, as my patching of existing code skills are dramatically better than my new-coding skills, and I run gentoo so already build everything from sources. I finally got tired of the kernel not defaulting to noatime, and hacked up a patch to do it, so I could remove noatime from all my fstab entries, for instance. But I already use a mkfs.btrfs helper script that adds options I prefer, so in this particular case, simply adding --mixed under appropriate conditions in the script is a far easier personal solution.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#7 lore raw
In reply to Christoph Anton Mitterer
Quoted 206 lines · Christoph Anton Mitterer

Hey.

I just repeatedly did the following twice on a ~8GB USB stick, under Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).

First, created some GPT on the stick:

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048575   511.0 MiB   EF02  BIOS boot partition
   2         1048576         3145727   1024.0 MiB  8300  Linux filesystem
   3         3145728         4194303   512.0 MiB   8300  Linux filesystem
   4         4194304         6291455   1024.0 MiB  8300  Linux filesystem
   5         6291456        15687644   4.5 GiB     8300  Linux filesystem

Then filesystems:

root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Label:              boot-main
UUID:               150ee9fb-c650-4b5b-8f64-606e589e429a
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              59.19MiB
  System:           DUP              12.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb2
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Label:              boot-data
UUID:               b1c1fc77-c965-4f0c-a2b5-44a301fd8772
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              59.19MiB
  System:           DUP              12.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb3
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive
/dev/sdb4
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Label:              boot-archive
UUID:               a063cf3b-0322-4ec7-a8c1-2dabecad9f57
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              59.19MiB
  System:           DUP              12.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  /dev/sdb4
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Label:              boot-rescue
UUID:               104b7bc3-3b8c-4a08-b0a6-9172ed664e68
Node size:          16384
Sector size:        4096
Filesystem size:    3.98GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP             211.75MiB
  System:           DUP              12.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     3.98GiB  /dev/sdb5

No errors in the kernel log.

Then I got usage info:

root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/
Overall:
    Device size:                   1.00GiB
    Device allocated:            126.38MiB
    Device unallocated:          897.62MiB
    Device missing:                  0.00B
    Used:                        256.00KiB
    Free (estimated):            905.62MiB      (min: 456.81MiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb3 8.00MiB

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb3 102.38MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb3 16.00MiB

Unallocated:

   /dev/sdb3     897.62MiB
root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/
Overall:
    Device size:                   1.00GiB
    Device allocated:            126.38MiB
    Device unallocated:          897.62MiB
    Device missing:                  0.00B
    Used:                        256.00KiB
    Free (estimated):            905.62MiB      (min: 456.81MiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb2 8.00MiB

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb2 102.38MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb2 16.00MiB

Unallocated:

   /dev/sdb2     897.62MiB
root@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/
Overall:
    Device size:                   3.98GiB
    Device allocated:            431.50MiB
    Device unallocated:            3.56GiB
    Device missing:                  0.00B
    Used:                        320.00KiB
    Free (estimated):              3.57GiB      (min: 1.79GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               16.00MiB      (used: 0.00B)

Data,single: Size:8.00MiB, Used:64.00KiB
/dev/sdb5 8.00MiB

Metadata,DUP: Size:203.75MiB, Used:112.00KiB
/dev/sdb5 407.50MiB

System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb5 16.00MiB

Unallocated:

   /dev/sdb5       3.56GiB

Still all fine.

Now I write to one of the filesystems (main at first, that still works) but then when I cp -a from another btrfs to data:
cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
cp: error writing ‘data/SHA512-sums’: No space left on device
cp: failed to extend ‘data/SHA512-sums’: No space left on device

(these files are just a few bytes large)
On the target fs, they're there but all 0 bytes large.

(btw: I've aliased cp to cp --reflink=auto

When I now repeat the usage info:

root@heisenberg:/data/SSS/boot# btrfs filesystem usage data
Overall:
    Device size:                   1.00GiB
    Device allocated:            118.38MiB
    Device unallocated:          905.62MiB
    Device missing:                  0.00B
    Used:                        256.00KiB
    Free (estimated):              8.00EiB      (min: 8.00EiB)
    Data ratio:                       -nan
    Metadata ratio:                   2.00
    Global reserve:               16.00MiB      (used: 0.00B)

Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdc3 102.38MiB

Quoted 8 lines

System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdc3 16.00MiB

Unallocated:

   /dev/sdc3     905.62MiB

It get's bogus... at least the Free space shows 8 EiB (if wishes were horses)...

You're likely hitting a bug introduced in kernel 4.2, and it got fixed
in 4.3 by: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=6af3e3adcac595a683bb55299a907d7d1ad61ab3

Because there are no data block groups you get the -nan data ratio there and the 8EiB, which is a problem in the tools that don't (or didn't, maybe they were fixed in the meanwhile) expect to find a filesystem without data block groups. Now what matters the most is the kernel bug, which is what leads to the ENOSPC errors when attempting to write data (even a 1 byte file).

Just try a 4.3 kernel, or a 4.2 kernel + that patch and confirm it fixes the issue.

thanks

Quoted 34 lines

Still no error in kernel logs:
3401.934514] BTRFS: device label boot-rescue devid 1 transid 8
/dev/sdc5

[ 3401.985858] BTRFS: device label boot-main devid 1 transid 8
/dev/sdc2
[ 3401.988601] BTRFS: device label boot-data devid 1 transid 9
/dev/sdc3
[ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9
/dev/sdc4
[ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled
[ 3403.721098] BTRFS: has skinny extents
[ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled
[ 3413.964042] BTRFS: has skinny extents
[ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled
[ 3522.902314] BTRFS: has skinny extents
[ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled
[ 3525.055659] BTRFS: has skinny extents

btrfs check doesn't show errors either.

After a balance, i can copy again, even though the usage still shows 8 EiB for a while...

Anyway how can it happen, that on a fresh btrfs with no files at all a balance is necessary?
Or is there some deeper corruption going on here?

Thanks,
Chris.

Signature · 5 lines
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."