why am I getting "No space left on device" here?
I'm no longer able to write to this btrfs filesystem:
# df -h /home Filesystem Size Used Avail Use% Mounted on /dev/sdb4 5.2T 3.6T 1.6T 71% /home
# btrfs fi show /home
Label: crawler-btrfs uuid: 60f1759c-45f6-4484-9f60-66a4e9bbf2b6
Total devices 2 FS bytes used 1.80TiB
devid 3 size 2.56TiB used 1.80TiB path /dev/sdb4
devid 4 size 2.56TiB used 1.80TiB path /dev/sda4Btrfs v3.12
# btrfs filesystem df /home Data, RAID1: total=1.75TiB, used=1.75TiB System, RAID1: total=32.00MiB, used=268.00KiB Metadata, RAID1: total=53.00GiB, used=51.71GiB
However:
# dd if=/dev/urandom of=bigfile dd: writing to `bigfile': No space left on device 186+0 records in 185+0 records out 94720 bytes (95 kB) copied, 0.0144045 s, 6.6 MB/s
I don't understand why - can anyone explain?
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Tomasz Chmielewski posted on Wed, 15 Jan 2014 11:55:43 +0100 as excerpted:
Quoted 5 lines
I'm no longer able to write to this btrfs filesystem:
# df -h /home Filesystem Size Used Avail Use% Mounted on /dev/sdb4 5.2T 3.6T 1.6T 71% /home
FWIW, standard df doesn't really know how to work with btrfs' advanced layout yet, so its output is, let's say "less than ideal", on btrfs, particularly on the various btrfs multi-device configurations.
btrfs fi show and btrfs fi df, combined, form the usable replacement on btrfs. Fortunately you listed their output as well...
Quoted 7 lines
# btrfs fi show /home Label: crawler-btrfs uuid: 60f1759c-45f6-4484-9f60-66a4e9bbf2b6 Total devices 2 FS bytes used 1.80TiB devid 3 size 2.56TiB used 1.80TiB path /dev/sdb4 devid 4 size 2.56TiB used 1.80TiB path /dev/sda4Btrfs v3.12
Looks pretty reasonable and well balanced, as a raid1 should be. =:^)
Only 1.80 TiB of 2.56 TiB on each device allocated, so there's plenty of room left to allocate additional chunks as needed. =:^)
Quoted 4 lines
# btrfs filesystem df /home Data, RAID1: total=1.75TiB, used=1.75TiB System, RAID1: total=32.00MiB, used=268.00KiB Metadata, RAID1: total=53.00GiB, used=51.71GiB
Data chunks are full, 1.75 TiB of 1.75 TiB, so it'll need to allocate a new data chunk pretty quickly when you start copying. (raid1 mode, so it'd allocate chunks in pairs on the two devices). FWIW, data chunks are 1 GiB each.
Metadata chunks, 51.71 GiB used of 53.00 GiB. 1.25+ GiB free. Metadata chunks are a quarter GiB (256 MiB) each, so that's several chunks worth, free.
Quoted 10 lines
However:
# dd if=/dev/urandom of=bigfile dd: writing to `bigfile': No space left on device 186+0 records in 185+0 records out 94720 bytes (95 kB) copied, 0.0144045 s, 6.6 MB/sI don't understand why - can anyone explain?
Well, there's two levels of explanation here, but unfortunately they don't fully cover it. Still, here's what's available:
At the first level, as hinted above in the df comments, btrfs' space calculation is MUCH more complex than that of a normal filesystem.
First, unlike a normal filesystem, btrfs data and metadata are treated separately, and they're very unlikely to run out together, so one or the other will be out while the other has room left. Then there's the fact that metadata is dup by default, while data is single, so metadata by default takes up twice the space it normally would. (Plus of course btrfs has checksums and even small partial-block file-tail data in its metadata, in addition to it all being dupped, so there's a lot MORE metadata to deal with on btrfs, than on a normal filesystem.)
In ordered to deal with that, btrfs sets up the empty filesystem as a big reserve pool of potential chunks that can be allocated to data or metadata as needed, so there's the whole already allocated vs. still unallocated and free to allocate thing, as displayed by btrfs fi show, that other filesystems don't normally deal with. Meanwhile, btrfs fi df displays, separately for each of data, metadata and system chunks, how much of the already allocated space is actually used. You can see my comments on your output above.
Then there's the whole multi-device thing and the various raid modes that btrfs has, that simply don't apply to normal filesystems. Both data/ metadata as raid1 with two devices is actually rather simple, since one copy goes to each device. Actually, that's even simpler than the default single-device case, since a single device defaults to dup metadata, single data, which is harder to figure out than a two-device raid1's simple one copy to each device rule. But a two-device raid1 is the simple case!
Then there's the fact that eventually, the plan is to allow different subvolumes to be configurable with different raid levels, so it could well be that you'd have raid1, raid10, raid6, raid5, and single, all on the same filesystem!
No *WONDER* df doesn't know how to report all this! Actually, they're already working on making df better for the simple all-one-type cases at least, but I doubt it'll ever be "good" at reporting for btrfs in the complex cases, since it's simply too simple a tool for that job.
This is actually covered on the btrfs wiki in the FAQ as well, altho I think I covered it more thoroughly above. But they'll give you some hints for dealing with the problem as well, and I'd definitely recommend spending some time reading the wiki in any case, since there's certainly more there that you're likely to find very useful as an admin running btrfs on your systems.
FAQ (space-related, see 1.3 and 1.4, and 4.4 thru 4.10)
https://btrfs.wiki.kernel.org/index.php/FAQ
General btrfs wiki link (bookmark it! =:^)
https://btrfs.wiki.kernel.org
The space-related FAQ entries should cover the theory, and give you some hints for fixing the problem as well, but there appears to be more going on in your case, as you have _PLENTY_ of unallocated space remaining so allocating more shouldn't be a problem. And I had a similar issue recently as well -- plenty of space left (tho in my case it was on a small mixed-mode filesystem).
That's the second level which I alluded to, where the FAQ and the answers above don't really cover things.
In my case, I was copying over a bunch of files at once. Actually, I had just done a fresh mkfs.btrfs on the /boot on one of my two ssds (with the other one still bootable in case something went wrong while I was setting up the new /boot, of course), and was trying to install grub2's modules and config files to it once again. As I said, that's a small (sub 1-gig so mixed-mode instead of separate data/metadata) filesystem, and the files in question were pretty small, too
But what I found here, was that while some files copied just fine, others failed. HOWEVER, I was using mc (aka midnight commander), and I used its directory diff feature to figure out what had copied and what hadn't, which left all the uncopied files in the source selected, so I could try copying them once again.
And the weird thing was, while the original copy errored out due with a no-space error, when I tried again to copy the files that hadn't copied, more of them copied over without error! By doing this a couple times, I was able to get everything copied over.
What happened was this. When the error occurred, while I had unallocated space left as shown in btrfs fi show, btrfs fi df showed nearly full usage. (Again, with a sub-gig filesystem, btrfs uses mixed-mode by default, so data/metadata combine, so it was just the single mixed-type chunk that was about full, not separate data/metadata. And a balance as suggested in the FAQ... didn't help, and I mount with compress=lzo,autodefrag already, and it was a fresh filesystem, so...)
But by trying the file copies again for just the files that had been missed the first time, the order was different, and something, somehow, triggered a new chunk allocation, that for whatever reason, btrfs had failed to allocate when it should have, the first time around.
Which brings us back to your case. While I was dealing with a small sub- gig filesystem and thus mixed-mode, you're dealing with a large filesystem and separate data/metadata chunks.
But just as my already allocated mixed-mode chunks were just about full and I needed another one allocated to complete the job, so your data chunks are full or very close, according to btrfs fi df, and you need a new one allocated (and if the file is greater than a gig in size, likely more than one) to finish the job.
And in both our cases, there's plenty of unallocated space in the pool, but for whatever reason, btrfs isn't allocating that new chunk when it should! Why, I can't say, but as I mentioned, I was able to work around the problem here by trying the remaining files in a different order, and at some point, btrfs figured out it needed that new chunk allocated, and everything went fine after that.
So... why btrfs is failing to allocate a new chunk when it needs to I can't say, but I *CAN* say you're not the only one to have run into the problem recently; I did too.
And just as I did, here, with a bit of monkeying around, you can /probably/ get btrfs to allocate that new data chunk and get on with things. But the trouble is, since I don't know what the exact problem is or what exactly I did to persuade btrfs to do that new chunk allocation, I can't tell you exactly what to do to get it to happen, all I can do is suggest you try copying smaller files or files of different sizes around a bit, hoping to trigger that allocation.
Once that chunk allocation happens, you should be good for at least a gig, since that's the data-chunk size, but if your file is over a gig in size, you may run into the problem again. In that case... Well, you could try copying several gigs of smaller files, then once it allocates what you need, delete them, leaving the data chunks allocated but with enough unused space to copy the original multi-gig file over.
But there's certainly some sort of chunk allocation bug involved here, since there was for me and is for you certainly unallocated space available to allocate new chunks, and from the btrfs fi df, we can see that the existing chunks are full and a new chunk SHOULD be allocated, but isn't being allocated, thus the bug. It'll probably be fixed in time, but meanwhile, try monkeying around a bit with other file sizes to hopefully work around the issue.
HTH =:^)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Am Mittwoch, 15. Januar 2014, 19:05:41 schrieb Duncan:
Quoted 25 lines
But just as my already allocated mixed-mode chunks were just about full and I needed another one allocated to complete the job, so your data chunks are full or very close, according to btrfs fi df, and you need a new one allocated (and if the file is greater than a gig in size, likely more than one) to finish the job.
And in both our cases, there's plenty of unallocated space in the pool, but for whatever reason, btrfs isn't allocating that new chunk when it should! Why, I can't say, but as I mentioned, I was able to work around the problem here by trying the remaining files in a different order, and at some point, btrfs figured out it needed that new chunk allocated, and everything went fine after that.
So... why btrfs is failing to allocate a new chunk when it needs to I can't say, but I *CAN* say you're not the only one to have run into the problem recently; I did too.
And just as I did, here, with a bit of monkeying around, you can /probably/ get btrfs to allocate that new data chunk and get on with things. But the trouble is, since I don't know what the exact problem is or what exactly I did to persuade btrfs to do that new chunk allocation, I can't tell you exactly what to do to get it to happen, all I can do is suggest you try copying smaller files or files of different sizes around a bit, hoping to trigger that allocation.
Quoted 6 lines
Once that chunk allocation happens, you should be good for at least a gig, since that's the data-chunk size, but if your file is over a gig in size, you may run into the problem again. In that case... Well, you could try copying several gigs of smaller files, then once it allocates what you need, delete them, leaving the data chunks allocated but with enough unused space to copy the original multi-gig file over.
I think you can cause BTRFS to allocate a chunk or more by creating a file with the fallocate command. Unless BTRFS doesn´t do it due to a bug and errors out with no space left.
fallocate gives also an easy way to find out how much it might still allocate and at what point it fails – and that without writing tons of data first. fallocate just triggers allocation and does not write any actual data.
Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
Martin Steigerwald posted on Wed, 15 Jan 2014 20:40:29 +0100 as excerpted:
Quoted 14 lines
Am Mittwoch, 15. Januar 2014, 19:05:41 schrieb Duncan:
Quoted 6 lines
But just as my already allocated mixed-mode chunks were just about full and I needed another one allocated to complete the job, so your data chunks are full or very close, according to btrfs fi df, and you need a new one allocated (and if the file is greater than a gig in size, likely more than one) to finish the job.
Quoted 6 lines
And in both our cases, there's plenty of unallocated space in the pool, but for whatever reason, btrfs isn't allocating that new chunk when it should! Why, I can't say, but as I mentioned, I was able to work around the problem here by trying the remaining files in a different order, and at some point, btrfs figured out it needed that new chunk allocated, and everything went fine after that.
Quoted 8 lines
I think you can cause BTRFS to allocate a chunk or more by creating a file with the fallocate command. Unless BTRFS doesn´t do it due to a bug and errors out with no space left.
fallocate gives also an easy way to find out how much it might still allocate and at what point it fails – and that without writing tons of data first. fallocate just triggers allocation and does not write any actual data.
Good idea. =:^)
I don't have the problem ATM so I can't test it, but hopefully Tomasz can.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 24 lines · Tomasz Chmielewski
I'm no longer able to write to this btrfs filesystem:
# df -h /home Filesystem Size Used Avail Use% Mounted on /dev/sdb4 5.2T 3.6T 1.6T 71% /home# btrfs fi show /home Label: crawler-btrfs uuid: 60f1759c-45f6-4484-9f60-66a4e9bbf2b6 Total devices 2 FS bytes used 1.80TiB devid 3 size 2.56TiB used 1.80TiB path /dev/sdb4 devid 4 size 2.56TiB used 1.80TiB path /dev/sda4Btrfs v3.12
# btrfs filesystem df /home Data, RAID1: total=1.75TiB, used=1.75TiB System, RAID1: total=32.00MiB, used=268.00KiB Metadata, RAID1: total=53.00GiB, used=51.71GiBHowever:
Quoted 8 lines
# dd if=/dev/urandom of=bigfile dd: writing to `bigfile': No space left on device 186+0 records in 185+0 records out 94720 bytes (95 kB) copied, 0.0144045 s, 6.6 MB/sI don't understand why - can anyone explain?
What kernel version? Can you:
umount
dmesg -n7
mount
And then try to reproduce the behavior and note any kernel messages in dmesg?
Chris Murphy
Quoted 9 lines
Quoted 8 lines
# dd if=/dev/urandom of=bigfile dd: writing to `bigfile': No space left on device 186+0 records in 185+0 records out 94720 bytes (95 kB) copied, 0.0144045 s, 6.6 MB/sI don't understand why - can anyone explain?
Quoted 8 lines
What kernel version? Can you:
umount
dmesg -n7
mountAnd then try to reproduce the behavior and note any kernel messages in dmesg?
It was running 3.13.0-rc6 at the time and there were no extra warnings etc. in dmesg.
I've removed a ~12 GB file (after which I was able to add new data) and rebooted since then, so unfortunately I won't be able to reproduce it for now.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 8 lines · On Wed, 15 Jan 2014 21:22:08 +0100 Tomasz Chmielewski <[email protected]> wrote:
Quoted 8 lines
What kernel version? Can you:
umount
dmesg -n7
mountAnd then try to reproduce the behavior and note any kernel messages in dmesg?
Turns out it's reproducible, with 3.13-rc8.
After reboot, I've started balance with:
# time btrfs fi balance start /home ERROR: error during balancing '/home' - Read-only file system There may be more info in syslog - try dmesg | tail
real 2434m32.786s user 0m0.000s sys 122m1.070s
dmesg shows:
(...)
Jan 17 17:04:35 test-5 kernel: [147299.097715] btrfs: found 79279 extents Jan 17 17:04:50 test-5 kernel: [147314.344600] btrfs: found 79279 extents Jan 17 17:04:51 test-5 kernel: [147314.888856] btrfs: relocating block group 4787911262208 flags 17 Jan 17 17:05:00 test-5 kernel: [147324.159656] btrfs: found 103218 extents Jan 17 17:05:17 test-5 kernel: [147341.085950] btrfs: found 103218 extents Jan 17 17:05:18 test-5 kernel: [147341.810090] btrfs: relocating block group 4786837520384 flags 17 Jan 17 17:05:27 test-5 kernel: [147351.174785] btrfs: found 70313 extents Jan 17 17:06:22 test-5 kernel: [147406.102437] btrfs: found 70313 extents Jan 17 17:07:40 test-5 kernel: [147483.489962] btrfs: relocating block group 4785763778560 flags 20 Jan 17 17:08:31 test-5 kernel: [147534.927240] BTRFS debug (device sdb4): run_one_delayed_ref returned -28 Jan 17 17:08:31 test-5 kernel: [147534.927244] ------------[ cut here ]------------ Jan 17 17:08:31 test-5 kernel: [147534.927260] BTRFS debug (device sdb4): run_one_delayed_ref returned -28 Jan 17 17:08:31 test-5 kernel: [147534.927262] BTRFS error (device sdb4) in btrfs_run_delayed_refs:2730: errno=-28 No space left Jan 17 17:08:31 test-5 kernel: [147534.927262] BTRFS info (device sdb4): forced readonly Jan 17 17:08:31 test-5 kernel: [147534.927465] WARNING: CPU: 0 PID: 2248 at fs/btrfs/super.c:254 __btrfs_abort_transaction+0x4d/0xff [btrfs]() Jan 17 17:08:31 test-5 kernel: [147534.927571] btrfs: Transaction aborted (error -28) Jan 17 17:08:31 test-5 kernel: [147534.927572] Modules linked in: tun xt_tcpudp iptable_filter ip_tables x_tables cpufreq_powersave cpufreq_ondemand cpufreq_conservative cpufreq_stats ipv6 btrfs zlib_deflate ext3 jbd loop i2c_i801 i2c_core pcspkr ehci_pci video button ehci_hcd lpc_ich mfd_core acpi_cpufreq ext4 crc16 jbd2 mbcache aacraid 3w_9xxx 3w_xxxx raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 sata_nv sata_sil sata_via sg sd_mod ahci libahci libata scsi_mod r8169 mii Jan 17 17:08:31 test-5 kernel: [147534.927987] CPU: 0 PID: 2248 Comm: btrfs-transacti Not tainted 3.13.0-rc8 #2 Jan 17 17:08:31 test-5 kernel: [147534.928043] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1106 10/17/2011 Jan 17 17:08:31 test-5 kernel: [147534.928146] 0000000000000009 ffff88080dbd1c48 ffffffff813898b7 0000000000000006 Jan 17 17:08:31 test-5 kernel: [147534.928249] ffff88080dbd1c98 ffff88080dbd1c88 ffffffff810370a9 ffff88080dbd1d80 Jan 17 17:08:31 test-5 kernel: [147534.928351] ffffffffa02d8524 00000000ffffffe4 ffff880813301000 ffff880036f5a960 Jan 17 17:08:31 test-5 kernel: [147534.928454] Call Trace: Jan 17 17:08:31 test-5 kernel: [147534.928506] [<ffffffff813898b7>] dump_stack+0x46/0x58 Jan 17 17:08:31 test-5 kernel: [147534.928561] [<ffffffff810370a9>] warn_slowpath_common+0x77/0x91 Jan 17 17:08:31 test-5 kernel: [147534.928618] [<ffffffffa02d8524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs] Jan 17 17:08:31 test-5 kernel: [147534.928721] [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43 Jan 17 17:08:31 test-5 kernel: [147534.928777] [<ffffffffa02d8524>] __btrfs_abort_transaction+0x4d/0xff [btrfs] Jan 17 17:08:31 test-5 kernel: [147534.928883] [<ffffffffa02ed6ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs] Jan 17 17:08:31 test-5 kernel: [147534.928945] [<ffffffffa02faaef>] btrfs_commit_transaction+0x70/0x7df [btrfs] Jan 17 17:08:31 test-5 kernel: [147534.929051] [<ffffffffa02f9345>] transaction_kthread+0xef/0x1c2 [btrfs] Jan 17 17:08:31 test-5 kernel: [147534.929112] [<ffffffffa02f9256>] ? open_ctree+0x1ac7/0x1ac7 [btrfs] Jan 17 17:08:31 test-5 kernel: [147534.929169] [<ffffffff8104ee9a>] kthread+0xcd/0xd5 Jan 17 17:08:31 test-5 kernel: [147534.929221] [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43 Jan 17 17:08:31 test-5 kernel: [147534.929278] [<ffffffff8138e6fc>] ret_from_fork+0x7c/0xb0 Jan 17 17:08:31 test-5 kernel: [147534.929333] [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43 Jan 17 17:08:31 test-5 kernel: [147534.929389] ---[ end trace 117a714822a6415b ]--- Jan 17 17:08:31 test-5 kernel: [147534.929442] BTRFS error (device sdb4) in btrfs_run_delayed_refs:2730: errno=-28 No space left
<---- unmounted, mounted with -o skip_balance here
Jan 18 01:00:47 test-5 kernel: [175855.856374] btrfs: device label crawler-btrfs devid 4 transid 401054 /dev/sda4 Jan 18 01:00:47 test-5 kernel: [175855.856616] btrfs: device label crawler-btrfs devid 3 transid 401054 /dev/sdb4 Jan 18 01:00:47 test-5 kernel: [175855.856905] btrfs: device label crawler-btrfs devid 3 transid 401054 /dev/sdb4 Jan 18 01:00:47 test-5 kernel: [175855.857613] btrfs: force zlib compression Jan 18 01:00:47 test-5 kernel: [175855.857681] btrfs: disk space caching is enabled Jan 18 01:01:07 test-5 kernel: [175876.239288] BTRFS debug (device sdb4): unlinked 1 orphans Jan 18 01:01:07 test-5 kernel: [175876.281954] btrfs: force skipping balance
# btrfs fi show /home
Label: crawler-btrfs uuid: 60f1759c-45f6-4484-9f60-66a4e9bbf2b6
Total devices 2 FS bytes used 1.76TiB
devid 3 size 2.56TiB used 1.76TiB path /dev/sdb4
devid 4 size 2.56TiB used 1.76TiB path /dev/sda4Btrfs v3.12
# btrfs fi df /home Data, RAID1: total=1.71TiB, used=1.71TiB System, RAID1: total=32.00MiB, used=260.00KiB Metadata, RAID1: total=53.00GiB, used=51.97GiB
# df -h /home Filesystem Size Used Avail Use% Mounted on /dev/sdb4 5.2T 3.6T 1.6T 69% /home
# btrfs device stats /home [/dev/sdb4].write_io_errs 0 [/dev/sdb4].read_io_errs 0 [/dev/sdb4].flush_io_errs 0 [/dev/sdb4].corruption_errs 0 [/dev/sdb4].generation_errs 0 [/dev/sda4].write_io_errs 0 [/dev/sda4].read_io_errs 0 [/dev/sda4].flush_io_errs 0 [/dev/sda4].corruption_errs 0 [/dev/sda4].generation_errs 0
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org