strange "No space left on device"
Hey.
I just repeatedly did the following twice on a ~8GB USB stick, under Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).
First, created some GPT on the stick:
Number Start (sector) End (sector) Size Code Name
1 2048 1048575 511.0 MiB EF02 BIOS boot partition 2 1048576 3145727 1024.0 MiB 8300 Linux filesystem 3 3145728 4194303 512.0 MiB 8300 Linux filesystem 4 4194304 6291455 1024.0 MiB 8300 Linux filesystem 5 6291456 15687644 4.5 GiB 8300 Linux filesystem
Then filesystems:
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.
Label: boot-main
UUID: 150ee9fb-c650-4b5b-8f64-606e589e429a Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no
Incompat features: extref, skinny-metadata Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb2
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.
Label: boot-data
UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772 Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no
Incompat features: extref, skinny-metadata Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb3
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive /dev/sdb4 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.
Label: boot-archive
UUID: a063cf3b-0322-4ec7-a8c1-2dabecad9f57 Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no
Incompat features: extref, skinny-metadata Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb4
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.
Label: boot-rescue
UUID: 104b7bc3-3b8c-4a08-b0a6-9172ed664e68 Node size: 16384
Sector size: 4096
Filesystem size: 3.98GiB
Block group profiles:
Data: single 8.00MiB Metadata: DUP 211.75MiB System: DUP 12.00MiB SSD detected: no
Incompat features: extref, skinny-metadata Number of devices: 1
Devices:
ID SIZE PATH
1 3.98GiB /dev/sdb5
No errors in the kernel log.
Then I got usage info:
root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/ Overall: Device size: 1.00GiB Device allocated: 126.38MiB Device unallocated: 897.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 905.62MiB (min: 456.81MiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)
Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb3 8.00MiB
Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb3 102.38MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb3 16.00MiB
Unallocated:
/dev/sdb3 897.62MiB
root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/ Overall: Device size: 1.00GiB Device allocated: 126.38MiB Device unallocated: 897.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 905.62MiB (min: 456.81MiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)
Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb2 8.00MiB
Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb2 102.38MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb2 16.00MiB
Unallocated:
/dev/sdb2 897.62MiB
root@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/ Overall: Device size: 3.98GiB Device allocated: 431.50MiB Device unallocated: 3.56GiB Device missing: 0.00B Used: 320.00KiB Free (estimated): 3.57GiB (min: 1.79GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)
Data,single: Size:8.00MiB, Used:64.00KiB
/dev/sdb5 8.00MiB
Metadata,DUP: Size:203.75MiB, Used:112.00KiB
/dev/sdb5 407.50MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb5 16.00MiB
Unallocated:
/dev/sdb5 3.56GiB
Still all fine.
Now I write to one of the filesystems (main at first, that still works) but then when I cp -a from another btrfs to data:
cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
cp: error writing ‘data/SHA512-sums’: No space left on device
cp: failed to extend ‘data/SHA512-sums’: No space left on device
(these files are just a few bytes large)
On the target fs, they're there but all 0 bytes large.
(btw: I've aliased cp to cp --reflink=auto
When I now repeat the usage info:
root@heisenberg:/data/SSS/boot# btrfs filesystem usage data Overall: Device size: 1.00GiB Device allocated: 118.38MiB Device unallocated: 905.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 8.00EiB (min: 8.00EiB) Data ratio: -nan Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)
Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdc3 102.38MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdc3 16.00MiB
Unallocated:
/dev/sdc3 905.62MiB
It get's bogus... at least the Free space shows 8 EiB (if wishes were horses)...
Still no error in kernel logs:
3401.934514] BTRFS: device label boot-rescue devid 1 transid 8 /dev/sdc5
[ 3401.985858] BTRFS: device label boot-main devid 1 transid 8 /dev/sdc2 [ 3401.988601] BTRFS: device label boot-data devid 1 transid 9 /dev/sdc3 [ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9 /dev/sdc4 [ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled [ 3403.721098] BTRFS: has skinny extents [ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled [ 3413.964042] BTRFS: has skinny extents [ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled [ 3522.902314] BTRFS: has skinny extents [ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled [ 3525.055659] BTRFS: has skinny extents
btrfs check doesn't show errors either.
After a balance, i can copy again, even though the usage still shows 8 EiB for a while...
Anyway how can it happen, that on a fresh btrfs with no files at all a balance is necessary?
Or is there some deeper corruption going on here?
Thanks,
Chris.
Quoted 5 lines · Christoph Anton Mitterer
Hey.
I just repeatedly did the following twice on a ~8GB USB stick, under Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).
Quoted 7 lines
First, created some GPT on the stick:
Number Start (sector) End (sector) Size Code Name
1 2048 1048575 511.0 MiB EF02 BIOS boot partition 2 1048576 3145727 1024.0 MiB 8300 Linux filesystem 3 3145728 4194303 512.0 MiB 8300 Linux filesystem 4 4194304 6291455 1024.0 MiB 8300 Linux filesystem 5 6291456 15687644 4.5 GiB 8300 Linux filesystem
These are all really small.
I would suggest running mkfs with --mixed for all of these
filesystems and trying again.
Hugo.
Quoted 234 lines
Then filesystems:
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Label: boot-main
UUID: 150ee9fb-c650-4b5b-8f64-606e589e429a Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no
Incompat features: extref, skinny-metadata Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb2root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Label: boot-data
UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772 Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no
Incompat features: extref, skinny-metadata Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb3root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive /dev/sdb4 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Label: boot-archive
UUID: a063cf3b-0322-4ec7-a8c1-2dabecad9f57 Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no
Incompat features: extref, skinny-metadata Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb4root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Label: boot-rescue
UUID: 104b7bc3-3b8c-4a08-b0a6-9172ed664e68 Node size: 16384
Sector size: 4096
Filesystem size: 3.98GiB
Block group profiles:
Data: single 8.00MiB Metadata: DUP 211.75MiB System: DUP 12.00MiB SSD detected: no
Incompat features: extref, skinny-metadata Number of devices: 1
Devices:
ID SIZE PATH
1 3.98GiB /dev/sdb5No errors in the kernel log.
Then I got usage info:
root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/ Overall: Device size: 1.00GiB Device allocated: 126.38MiB Device unallocated: 897.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 905.62MiB (min: 456.81MiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb3 8.00MiBMetadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb3 102.38MiBSystem,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb3 16.00MiBUnallocated:
/dev/sdb3 897.62MiBroot@heisenberg:/data/SSS/boot# btrfs filesystem usage main/ Overall: Device size: 1.00GiB Device allocated: 126.38MiB Device unallocated: 897.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 905.62MiB (min: 456.81MiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb2 8.00MiBMetadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb2 102.38MiBSystem,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb2 16.00MiBUnallocated:
/dev/sdb2 897.62MiBroot@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/ Overall: Device size: 3.98GiB Device allocated: 431.50MiB Device unallocated: 3.56GiB Device missing: 0.00B Used: 320.00KiB Free (estimated): 3.57GiB (min: 1.79GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)Data,single: Size:8.00MiB, Used:64.00KiB
/dev/sdb5 8.00MiBMetadata,DUP: Size:203.75MiB, Used:112.00KiB
/dev/sdb5 407.50MiBSystem,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb5 16.00MiBUnallocated:
/dev/sdb5 3.56GiBStill all fine.
Now I write to one of the filesystems (main at first, that still works) but then when I cp -a from another btrfs to data:
cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
cp: error writing ‘data/SHA512-sums’: No space left on device
cp: failed to extend ‘data/SHA512-sums’: No space left on device(these files are just a few bytes large)
On the target fs, they're there but all 0 bytes large.(btw: I've aliased cp to cp --reflink=auto
When I now repeat the usage info:
root@heisenberg:/data/SSS/boot# btrfs filesystem usage data Overall: Device size: 1.00GiB Device allocated: 118.38MiB Device unallocated: 905.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 8.00EiB (min: 8.00EiB) Data ratio: -nan Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdc3 102.38MiBSystem,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdc3 16.00MiBUnallocated:
/dev/sdc3 905.62MiBIt get's bogus... at least the Free space shows 8 EiB (if wishes were horses)...
Still no error in kernel logs:
3401.934514] BTRFS: device label boot-rescue devid 1 transid 8/dev/sdc5[ 3401.985858] BTRFS: device label boot-main devid 1 transid 8 /dev/sdc2 [ 3401.988601] BTRFS: device label boot-data devid 1 transid 9 /dev/sdc3 [ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9 /dev/sdc4 [ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled [ 3403.721098] BTRFS: has skinny extents [ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled [ 3413.964042] BTRFS: has skinny extents [ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled [ 3522.902314] BTRFS: has skinny extents [ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled [ 3525.055659] BTRFS: has skinny extentsbtrfs check doesn't show errors either.
After a balance, i can copy again, even though the usage still shows 8 EiB for a while...
Anyway how can it happen, that on a fresh btrfs with no files at all a balance is necessary?
Or is there some deeper corruption going on here?Thanks,
Chris.
Signature · 4 lines
Hugo Mills | Reintarnation: Coming back from the dead as a hugo@... carfax.org.uk | hillbilly http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 1 line · Hugo Mills
These are all really small.
Well enough for booting =)
Quoted 2 lines
I would suggest running mkfs with --mixed for all of these filesystems and trying again.
I thought btrfs would do that automatically:
https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_small
"Using mixed block groups is recommended for filesystems of 1GiB or smaller and mkfs.btrfs will force mixed block groups automatically in that case."
Anyway I even if that's the reason, than I think something's quite wrong here... a) it doesn't happen always (I just create 10 times a fs in the same partition, and no problem) ... b) in those cases where I could produce the issue, it went away after a balance... so perhaps they just just balance the fs right after mkfs.... o.O
Cheers,
Chris.
[...]
Quoted 1 line
3 3145728 4194303 512.0 MiB 8300 Linux filesystem
[...]
Quoted 4 lines
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.
Quoted 15 lines
Label: boot-data
UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772
Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 59.19MiB
System: DUP 12.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb3Somehow your mkfs.btrfs (or some other part of the system) assumed the wrong size for partition 3. To me, that sounds like a possible cause of the errors and 'EiB' reported size.
For RAID tests in VMs I have also been using 4-8GB disk sizes (instead of TB sized like the real hardware) and I encountered many 'No space left on device' message (with older kernels).
A quick test with kernel 4.3.0 now:
# dd if=/dev/zero of=sd.bin bs=1M count=512
# losetup -f sd.bin
# mkfs.btrfs --nodiscard --label boot-data /dev/loop0 SMALL VOLUME: forcing mixed metadata/data groups btrfs-progs v4.2.3+20151102 See http://btrfs.wiki.kernel.org for more information.
Label: boot-data
UUID: f152781b-b96b-4969-96b8-eede3e3dabc2
Node size: 4096
Sector size: 4096
Filesystem size: 512.00MiB
Block group profiles:
Data+Metadata: single 8.00MiB
System: single 4.00MiB
SSD detected: no
Incompat features: mixed-bg, extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 512.00MiB /dev/loop0Henk Slager posted on Sun, 08 Nov 2015 19:18:19 +0100 as excerpted:
Quoted 22 lines
[...]
Quoted 1 line
3 3145728 4194303 512.0 MiB 8300 Linux filesystem[...]
Quoted 4 lines
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Quoted 14 lines
Label: boot-data UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772 Node size: 16384 Sector size: 4096 Filesystem size: 1.00GiB Block group profiles: Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 1.00GiB /dev/sdb3
Quoted 3 lines
Somehow your mkfs.btrfs (or some other part of the system) assumed the wrong size for partition 3. To me, that sounds like a possible cause of the errors and 'EiB' reported size.
Wow, yes! Good catch, Henk! =:^) Hugo obviously didn't catch it, and I wouldn't have either, as the bad size detection behavior is so unexpected, it just wouldn't occur to me to look!
Something's clearly wrong, and as it could clearly either overwrite the next partition(s) if the partitioning detection is at fault and doesn't stop it, or throw all sorts of monkey wrenches into btrfs behavior if it does, this is rather alarming behavior indeed! Clearly we need devs to take a look and figure out what bugged out, and how to fix it.
That potentially explains why you got separate data/metadata instead of mixed mode, and why it worked most of the time but failed occasionally, since presumably when it worked the size detection worked as well.
(Apparently, btrfs-progs-4.3 does away with the default to mixed-mode at 1 GiB or under, tho it is still recommended. I'm not exactly sure of why, tho I think it had to do with being able to use sub-GiB btrfs for testing without having to worry about mixed mode. Still, one would think, since it's recommended, that they'd simply provide an option to make it split mode, instead of changing the default to something that's not recommended. So from 4.3 userspace you'd have to specifically set mixed, but you're on 4.2.2 userspace, so that shouldn't be the case yet. And in any case that doesn't explain the very badly detected size and an attempt to create a 1 GiB btrfs on a 1/2 GiB partition, which is obviously the real problem, regardless of whatever layer it occurred in.)
Meanwhile, btrfs fi usage doesn't understand mixed-mode yet, and returns 8 EiB for some values if it is used, which is originally what I thought was happening. But the problem is elsewhere, as it wasn't mixed mode, but rather, a very /very/ screwed up size detection.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 4 lines · Duncan
Wow, yes! Good catch, Henk! =:^) Hugo obviously didn't catch it, and I
wouldn't have either, as the bad size detection behavior is so unexpected, it just wouldn't occur to me to look!
Hmm... all that *may* be more likely an error of myself when copying and pasting the terminal output together:
I did actually change the 3rd partition to use 1GiB in later tries at the expense of the 5th one shrinking, so the part table would have looked like this in these later tries:
512M
1G
1G
1G
4G
Which would again fit the output of the various mkfs.btrfs.
Sorry if that was the case, apologies for any confusion.
The problem seemed to went away when explicitly using --mixed.
Quoted 2 lines
(Apparently, btrfs-progs-4.3 does away with the default to mixed- mode at 1 GiB or under, tho it is still recommended.
Well I still had 4.2 ...
Quoted 4 lines
I'm not exactly sure of
why, tho I think it had to do with being able to use sub-GiB btrfs for
testing without having to worry about mixed mode.
Kinda strange... shouldn't it work out of the box for users and not developers?
To be honest, no one should need to read through the wiki, just to be able to create a small sized fs.
And even if no mixed D/M block group allocation was used... it shouldn't just fail out-of-the-box with a few byte large files on a 1 GB fs.
Cheers,
Chris.
Christoph Anton Mitterer posted on Sun, 08 Nov 2015 23:10:57 +0100 as excerpted:
Quoted 11 lines · Duncan
Quoted 3 lines
Wow, yes! Good catch, Henk! =:^) Hugo obviously didn't catch it, and I wouldn't have either, as the bad size detection behavior is so unexpected, it just wouldn't occur to me to look!
Hmm... all that *may* be more likely an error of myself when copying and pasting the terminal output together:
I did actually change the 3rd partition to use 1GiB in later tries at the expense of the 5th one shrinking, so the part table would have looked like this in these later tries:
512M 1G 1G 1G 4G
Oops! =:^(
Quoted 6 lines
Which would again fit the output of the various mkfs.btrfs.
Sorry if that was the case, apologies for any confusion.
The problem seemed to went away when explicitly using --mixed.
Good. Whatever the cause of it not doing mixed by default (perhaps it's an off-by-one error and filesystems of exactly 1 GiB weren't defaulting to mixed as expected), the fact that mixed was the default on small filesystems would mean separate data/metadata wouldn't have gotten the same full testing coverage, so maybe there's a bug... In which case the 4.3 change discussed below would certainly tend to trigger it more often, hopefully getting it fixed. =:^)
Quoted 4 lines
Quoted 2 lines
(Apparently, btrfs-progs-4.3 does away with the default to mixed- mode at 1 GiB or under, tho it is still recommended.
Well I still had 4.2 ...
That's why I had that as a parenthetical. It didn't apply to your current case, but could to others, and I've learned to write with both the lurkers and the googlers who will happen on my post later, in mind.
Quoted 9 lines
Quoted 3 lines
I'm not exactly sure of why, tho I think it had to do with being able to use sub-GiB btrfs for testing without having to worry about mixed mode.
Kinda strange... shouldn't it work out of the box for users and not developers?
To be honest, no one should need to read through the wiki, just to be able to create a small sized fs.
That's why, tho I didn't press it, I asked for clarification when I saw it mentioned on the btrfs-progs 4.3-rc1 announcement, verifying that mixed was still recommended. Chris Murphy (another non-btrfs-dev list regular, I think he works or volunteers for one of the distros) questioned it as well, on the same announcement.
But not being able to personally show-them-the-code, as long as there's at least an option to do it the way I want (which there is, the already mentioned --mixed option), I've learned not to press it.[1]
Quoted 2 lines
And even if no mixed D/M block group allocation was used... it shouldn't just fail out-of-the-box with a few byte large files on a 1 GB fs.
You are absolutely correct. There's a bug somewhere. Earlier I thought it was the size detection bug Henk spotted, since even if there's some other bug here it'd pale in comparision, but since that bug turned out to trace to your copy/paste, that isn't it, and we still have the much more ordinary level bug to deal with.
But as I said above, with progs 4.3 defaulting to unmixed even on small filesystems now, if indeed it is tied to unmixed on a small filesystem, that bug's very likely to get MUCH more common, and thus should be fixed relatively quickly, I'd guess by the time 4.4 comes out, given the history of other bugs that turn out to be quite common.
---
[1] Tho for my own systems I sometimes patch in the behavior as I think it should be, as my patching of existing code skills are dramatically better than my new-coding skills, and I run gentoo so already build everything from sources. I finally got tired of the kernel not defaulting to noatime, and hacked up a patch to do it, so I could remove noatime from all my fstab entries, for instance. But I already use a mkfs.btrfs helper script that adds options I prefer, so in this particular case, simply adding --mixed under appropriate conditions in the script is a far easier personal solution.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 206 lines · Christoph Anton Mitterer
Hey.
I just repeatedly did the following twice on a ~8GB USB stick, under Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).
First, created some GPT on the stick:
Number Start (sector) End (sector) Size Code Name 1 2048 1048575 511.0 MiB EF02 BIOS boot partition 2 1048576 3145727 1024.0 MiB 8300 Linux filesystem 3 3145728 4194303 512.0 MiB 8300 Linux filesystem 4 4194304 6291455 1024.0 MiB 8300 Linux filesystem 5 6291456 15687644 4.5 GiB 8300 Linux filesystemThen filesystems:
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Label: boot-main UUID: 150ee9fb-c650-4b5b-8f64-606e589e429a Node size: 16384 Sector size: 4096 Filesystem size: 1.00GiB Block group profiles: Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 1.00GiB /dev/sdb2root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Label: boot-data UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772 Node size: 16384 Sector size: 4096 Filesystem size: 1.00GiB Block group profiles: Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 1.00GiB /dev/sdb3root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive /dev/sdb4 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Label: boot-archive UUID: a063cf3b-0322-4ec7-a8c1-2dabecad9f57 Node size: 16384 Sector size: 4096 Filesystem size: 1.00GiB Block group profiles: Data: single 8.00MiB Metadata: DUP 59.19MiB System: DUP 12.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 1.00GiB /dev/sdb4root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information.Label: boot-rescue UUID: 104b7bc3-3b8c-4a08-b0a6-9172ed664e68 Node size: 16384 Sector size: 4096 Filesystem size: 3.98GiB Block group profiles: Data: single 8.00MiB Metadata: DUP 211.75MiB System: DUP 12.00MiB SSD detected: no Incompat features: extref, skinny-metadata Number of devices: 1 Devices: ID SIZE PATH 1 3.98GiB /dev/sdb5No errors in the kernel log.
Then I got usage info:
root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/ Overall: Device size: 1.00GiB Device allocated: 126.38MiB Device unallocated: 897.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 905.62MiB (min: 456.81MiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb38.00MiBMetadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb3102.38MiBSystem,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb316.00MiBUnallocated:
/dev/sdb3 897.62MiB root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/ Overall: Device size: 1.00GiB Device allocated: 126.38MiB Device unallocated: 897.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 905.62MiB (min: 456.81MiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb28.00MiBMetadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb2102.38MiBSystem,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb216.00MiBUnallocated:
/dev/sdb2 897.62MiB root@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/ Overall: Device size: 3.98GiB Device allocated: 431.50MiB Device unallocated: 3.56GiB Device missing: 0.00B Used: 320.00KiB Free (estimated): 3.57GiB (min: 1.79GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)Data,single: Size:8.00MiB, Used:64.00KiB
/dev/sdb58.00MiBMetadata,DUP: Size:203.75MiB, Used:112.00KiB
/dev/sdb5407.50MiBSystem,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb516.00MiBUnallocated:
/dev/sdb5 3.56GiBStill all fine.
Now I write to one of the filesystems (main at first, that still works) but then when I cp -a from another btrfs to data:
cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
cp: error writing ‘data/SHA512-sums’: No space left on device
cp: failed to extend ‘data/SHA512-sums’: No space left on device(these files are just a few bytes large)
On the target fs, they're there but all 0 bytes large.(btw: I've aliased cp to cp --reflink=auto
When I now repeat the usage info:
root@heisenberg:/data/SSS/boot# btrfs filesystem usage data Overall: Device size: 1.00GiB Device allocated: 118.38MiB Device unallocated: 905.62MiB Device missing: 0.00B Used: 256.00KiB Free (estimated): 8.00EiB (min: 8.00EiB) Data ratio: -nan Metadata ratio: 2.00 Global reserve: 16.00MiB (used: 0.00B)Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdc3102.38MiB
Quoted 8 lines
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdc316.00MiBUnallocated:
/dev/sdc3 905.62MiBIt get's bogus... at least the Free space shows 8 EiB (if wishes were horses)...
You're likely hitting a bug introduced in kernel 4.2, and it got fixed
in 4.3 by: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=6af3e3adcac595a683bb55299a907d7d1ad61ab3
Because there are no data block groups you get the -nan data ratio there and the 8EiB, which is a problem in the tools that don't (or didn't, maybe they were fixed in the meanwhile) expect to find a filesystem without data block groups. Now what matters the most is the kernel bug, which is what leads to the ENOSPC errors when attempting to write data (even a 1 byte file).
Just try a 4.3 kernel, or a 4.2 kernel + that patch and confirm it fixes the issue.
thanks
Quoted 34 lines
Still no error in kernel logs:
3401.934514] BTRFS: device label boot-rescue devid 1 transid 8/dev/sdc5[ 3401.985858] BTRFS: device label boot-main devid 1 transid 8 /dev/sdc2 [ 3401.988601] BTRFS: device label boot-data devid 1 transid 9 /dev/sdc3 [ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9 /dev/sdc4 [ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled [ 3403.721098] BTRFS: has skinny extents [ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled [ 3413.964042] BTRFS: has skinny extents [ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled [ 3522.902314] BTRFS: has skinny extents [ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled [ 3525.055659] BTRFS: has skinny extentsbtrfs check doesn't show errors either.
After a balance, i can copy again, even though the usage still shows 8 EiB for a while...
Anyway how can it happen, that on a fresh btrfs with no files at all a balance is necessary?
Or is there some deeper corruption going on here?Thanks,
Chris.
Signature · 5 lines
Filipe David Manana, "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."