snapshot space available
Linux xev 3.10-2-amd64 #1 SMP Debian 3.10.5-1 (2013-08-07) x86_64 GNU/Linux
I've attached a file of script output from a system running the above Debian kernel on a system with an Intel 120G SSD. To get this working again I extended the filesystem in question to also use a small USB flash device (4G from memory) and then deleted some old snapshots to free some space.
The most intereting part of the script output is pasted below. The system reports that the filesystem is 82% full but that there is no space left.
root@xev:~# df -h / Filesystem Size Used Avail Use% Mounted on /dev/disk/by-uuid/586e6f48-2985-4115-9f89-f844b319c7c0 108G 87G 21G 82% / root@xev:~# btrfs filesystem df / Data: total=101.57GB, used=81.50GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=3.00GB, used=2.50GB Metadata: total=8.00MB, used=0.00 root@xev:~# ls -l > test bash: test: No space left on device root@xev:~# touch test touch: cannot touch ‘test’: No space left on device root@xev:~#
Signature · 2 lines
My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Script started on Fri 23 Aug 2013 00:19:03 EST
root@xev:~# df -h / Filesystem Size Used Avail Use% Mounted on /dev/disk/by-uuid/586e6f48-2985-4115-9f89-f844b319c7c0 108G 87G 21G 82% / root@xev:~# btrfs df /[1@f[1@i[1@l[1@e[1@s[1@y[1@s[1@t[1@e[1@m[1@ Data: total=101.57GB, used=81.50GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=3.00GB, used=2.50GB Metadata: total=8.00MB, used=0.00 root@xev:~# dmesg|tail -50 [ 15.128069] e1000e 0000:00:19.0: irq 43 for MSI/MSI-X [ 15.128149] IPv6: ADDRCONF(NETDEV_UP): mb0: link is not ready [ 15.129817] device eth0 entered promiscuous mode [ 15.145722] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 15.147178] device eth1 entered promiscuous mode [ 15.149120] 8139too 0000:04:01.0 eth1: link up, 100Mbps, full-duplex, lpa 0x41E1 [ 15.153676] xenbr0: port 3(eth1) entered listening state [ 15.153690] xenbr0: port 3(eth1) entered listening state [ 15.153697] xenbr0: port 2(eth0) entered listening state [ 15.153703] xenbr0: port 2(eth0) entered listening state [ 15.153735] IPv6: ADDRCONF(NETDEV_UP): xenbr0: link is not ready [ 15.660053] xenbr0: port 2(eth0) entered disabled state [ 17.372874] e1000e: mb0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [ 17.373017] IPv6: ADDRCONF(NETDEV_CHANGE): mb0: link becomes ready [ 17.373047] xenbr0: port 1(mb0) entered listening state [ 17.373054] xenbr0: port 1(mb0) entered listening state [ 18.160017] xenbr0: port 3(eth1) entered learning state [ 20.376017] xenbr0: port 1(mb0) entered learning state [ 21.168010] xenbr0: topology change detected, sending tcn bpdu [ 21.168035] xenbr0: port 3(eth1) entered forwarding state [ 21.168062] IPv6: ADDRCONF(NETDEV_CHANGE): xenbr0: link becomes ready [ 23.384013] xenbr0: topology change detected, sending tcn bpdu [ 23.384033] xenbr0: port 1(mb0) entered forwarding state [ 23.805317] RPC: Registered named UNIX socket transport module. [ 23.805321] RPC: Registered udp transport module. [ 23.805323] RPC: Registered tcp transport module. [ 23.805324] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 23.811142] FS-Cache: Loaded [ 23.812150] Key type dns_resolver registered [ 23.822368] FS-Cache: Netfs 'nfs' registered for caching [ 23.838200] Installing knfsd (copyright (C) 1996 [email protected]). [ 23.843440] SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts [ 24.342794] SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts [ 24.423047] auditd (2584): /proc/2584/oom_adj is deprecated, please use /proc/2584/oom_score_adj instead. [ 24.579080] lp0: using parport0 (interrupt-driven). [ 24.586862] ppdev: user-space parallel port driver [ 27.275382] BTRFS debug (device dm-0): truncated 2 orphans [ 62.198125] SELinux: initialized (dev proc, type proc), uses genfs_contexts [ 580.759753] NFS: Registering the id_resolver key type [ 580.759772] Key type id_resolver registered [ 580.759774] Key type id_legacy registered [ 595.768019] RPC: AUTH_GSS upcall timed out. [ 595.768019] Please check user daemon is running. [ 595.773813] SELinux: initialized (dev 0:27, type nfs4), uses genfs_contexts [ 595.779200] SELinux: initialized (dev 0:28, type nfs4), uses genfs_contexts [ 595.782249] SELinux: initialized (dev 0:29, type nfs4), uses genfs_contexts [ 606.227371] SELinux: initialized (dev 0:27, type nfs4), uses genfs_contexts [ 606.230441] SELinux: initialized (dev 0:28, type nfs4), uses genfs_contexts [ 606.233676] SELinux: initialized (dev 0:30, type nfs4), uses genfs_contexts [ 658.902288] BTRFS debug (device dm-0): unlinked 1 orphans root@xev:~# ls -l > test bash: test: No space left on device root@xev:~# echo =n [K[K[Kn[K[K[K[K[K[Ktouch test touch: cannot touch ‘test’: No space left on device root@xev:~# exit
Script done on Fri 23 Aug 2013 00:19:51 EST
You are out of metadata, not "normal space".
However, the good question is why 0.5GB of metadata are unused and btrfs reports no space left.
I have seen similar behaviour on a machine of mine, with exactly 0.5GB of metadata unused.
Regards
2013/8/26 Russell Coker <[email protected]>:
Quoted 30 lines
Linux xev 3.10-2-amd64 #1 SMP Debian 3.10.5-1 (2013-08-07)
x86_64GNU/LinuxI've attached a file of script output from a system running the above Debian kernel on a system with an Intel 120G SSD. To get this working again I extended the filesystem in question to also use a small USB flash device (4G from memory) and then deleted some old snapshots to free some space.
The most intereting part of the script output is pasted below. The system reports that the filesystem is 82% full but that there is no space left.
root@xev:~# df -h / Filesystem Size Used Avail Use% Mounted on /dev/disk/by-uuid/586e6f48-2985-4115-9f89-f844b319c7c0 108G 87G 21G 82% / root@xev:~# btrfs filesystem df / Data: total=101.57GB, used=81.50GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=3.00GB, used=2.50GB Metadata: total=8.00MB, used=0.00 root@xev:~# ls -l > test bash: test: No space left on device root@xev:~# touch test touch: cannot touch ‘test’: No space left on device root@xev:~#--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
Quoted 5 lines · Clemens Eisserer
You are out of metadata, not "normal space".
However, the good question is why 0.5GB of metadata are unused and btrfs reports no space left.
I have seen similar behaviour on a machine of mine, with exactly 0.5GB of metadata unused.
Since metadata is set to DUP, the numbers are actually 6GB allocated and 5GB used, and I'm guessing that at the time it starts to use the last chunk of allocated space, it also wants to allocate a new one. But it can't allocate more for metadata because all free space in the fs is already allocated for data chunks.
Presently btrfs seems unable to unwind itself from this situation without a manual balance, which in many instances also requires more free space than is available once you're in this situation. So I called it a filesystem wedged into a corner. Fortunately so far I've found it to be rather docile in accepting a small device to give it a little extra free space, do a balance, and then delete the device from the volume.
I haven't tried doing a partial balance, deleting the extra device, then finishing the balance.
But this suggestion is certainly non-obvious to the mortal and sane. Especially if you do a df, which bases its values off the used portion of allocated chunks, the file system is distinctly not out of free space. So the confusion is understandable.
Chris Murphy
Quoted 1 line · Chris Murphy
I haven't tried doing a partial balance, deleting the extra device, then finishing the balance.
Btrfs FAQ suggests a partial balance for this situation. So it's worth a try before adding a device.
Chris Murphy
How would I determine when this situation is approaching? On all important systems I have cron jobs and nagios checks to determine via df when thw file system is about to run out of space. How do I do the same for this condition?
Quoted 14 lines · Chris Murphy
Quoted 1 line · Chris Murphy
I haven't tried doing a partial balance, deleting the extra device,
then finishing the balance.
Btrfs FAQ suggests a partial balance for this situation. So it's worth a try before adding a device.
Chris Murphy
Signature · 1 line
Sent from my Samsung Galaxy Note 2 with K-9 Mail.
Quoted 1 line · Russell Coker
How would I determine when this situation is approaching? On all important systems I have cron jobs and nagios checks to determine via df when thw file system is about to run out of space. How do I do the same for this condition?
I don't think it's possible to infer this condition from a single command.
However, if you look at only 'btrfs fi show', a small difference between size and used could be used as a trigger for a partial balance operation to avoid the condition.
What's a small difference? I'd say at least 2GB, or the size of the maximum expected file size + 2GB. That way a single write operation doesn't end up causing all free space to be allocated for data, leaving no more free space to be allocated for metadata.
Chris Murphy
Quoted 12 lines · Russell Coker
root@xev:~# btrfs filesystem df / Data: total=101.57GB, used=81.50GB System, DUP: total=8.00MB, used=20.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=3.00GB, used=2.50GB Metadata: total=8.00MB, used=0.00 root@xev:~# ls -l > test bash: test: No space left on device root@xev:~# touch test touch: cannot touch ‘test’: No space left on device root@xev:~#
The issue is data chunks remaining allocated for data, and not enough free space to allocate metadata chunks. Ultimately, the volume needs to be balanced.
I'm uncertain if autodefrag avoids this problem. It does seem like in certain instances, like this, the file system needs to be able to prune itself somehow, like a partial balance to consolidate data chunks and then release their space so they can become metadata chunks.
Chris Murphy
Chris Murphy posted on Mon, 26 Aug 2013 09:15:46 -0600 as excerpted:
Quoted 4 lines
I'm uncertain if autodefrag avoids this problem. It does seem like in certain instances, like this, the file system needs to be able to prune itself somehow, like a partial balance to consolidate data chunks and then release their space so they can become metadata chunks.
Intriguing question re defrag. I hadn't thought of the possibility until you suggested it, but indeed, tracking hundreds of extents instead of one or a dozen, does sound like it could reduce metadata usage, reducing the chance of running into the issue in the first place as well as lessening the chance of a "simple" delete temporarily requiring significant new metadata resources in ordered to track all those extent frees before the final atomic root entry update.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman