Quota limit question
Hello there,
I'm trying out btrfs on a machine we use to host a number of
containers. After a misbehaved process filled the partition allocated to the containers, I decided to experiment with quotas to isolate the containers from each other. But I've now run into an oddity with one of the containers, which reports being out of space:
# btrfs qgroup limit 2000m 0/261 . && touch x
touch: cannot touch ‘x’: Disk quota exceededThe strange thing is that it doesn't seem to be actually out of space:
# btrfs qgroup show -p -r -e /var | grep 261
0/261 1111810048 391114752 2097152000 0 --- which pretty-printed is 1.04G rfer and 0.36G excl (perhaps the qgroup show command could take an option to display in other units?)
I can only get it to allow me to start using it again if I go over 5808M:
# btrfs qgroup limit 5807m 0/261 . && rm -f x && touch x
rm: cannot remove ‘x’: Disk quota exceeded
# btrfs qgroup limit 5808m 0/261 . && rm -f x && touch x
#Why specifically 5808 I'm not sure, but I binary searched until I got to that number. Does anyone have a clue as to why that might be happening, and perhaps what I'm missing?
For completeness, some details on the filesystem and system:
# btrfs fi show /var
Label: var uuid: 815b3280-e90f-483a-b244-1d2dfe9b6e67
Total devices 2 FS bytes used 31.48GiB
devid 1 size 80.00GiB used 55.91GiB path /dev/sda3
devid 2 size 80.00GiB used 55.91GiB path /dev/sdb3 root@riff:/var/lib/lxc/async-local-machine-2/rootfs# btrfs fi df /var
Data, RAID1: total=53.88GiB, used=30.45GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=2.00GiB, used=1.03GiB # btrfs qgroup show -p -r -e /var
qgroupid rfer excl max_rfer max_excl parent
-------- ---- ---- -------- -------- ------
0/5 1486852096 1252569088 0 0 ---
0/259 727175168 104947712 0 5368709120 ---
0/261 1111810048 391114752 2097152000 0 ---
0/265 1255923712 442871808 0 5368709120 ---
0/271 831856640 333189120 0 5368709120 ---
0/274 498761728 22827008 0 5368709120 ---
0/283 7666098176 6691426304 10737418240 0 ---
0/288 1118441472 348901376 0 5368709120 ---
0/289 11134029824 10498187264 16106127360 0 ---
0/290 1412505600 694210560 10737418240 0 ---
0/292 1131053056 333373440 0 5368709120 ---
0/293 1258176512 401141760 0 5368709120 ---
0/306 1430532096 656773120 0 5368709120 ---
0/318 9309212672 8509857792 10737418240 0 ---
0/320 860209152 837406720 0 5368709120 ---
0/323 1167962112 469741568 0 5368709120 --- # btrfs --version
Btrfs v3.12 # uname -a
Linux riff 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014
x86_64 x86_64 x86_64 GNU/LinuxThanks,
Signature · 2 lines
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko Async Open Source | [+55 16] 9 9112 6430 | http://launchpad.net/~kiko
Christian Robottom Reis posted on Tue, 16 Dec 2014 23:15:37 -0200 as excerpted:
Quoted 5 lines
# btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch ‘x’: Disk quota exceededThe strange thing is that it doesn't seem to be actually out of space:
Quoted 8 lines
# btrfs qgroup show -p -r -e /var | grep 261 0/261 1111810048 391114752 2097152000 0 ---# btrfs --version Btrfs v3.12# uname -a Linux riff 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Does the problem still exist with a current 3.18 series kernel and 3.17 series btrfs userspace? The quota code in the versions you appear to be running had some (rather severe, quotes could go negative at times!) known bugs and was rewritten recently. I don't use quotas here and don't know for sure whether the quota rewrite made it into kernel 3.17, but current 3.18 should have it, and current userspace is I believe 3.17.3, so you should consider upgrading it as well.
More in general, consider your use-case. While btrfs is no longer the experimental filesystem it once was (tho the warnings weren't removed until after the old 3.12/3.13 you're running, AFAIK), it's still maturing, and running year-old code still means running code with bugs that are known and have been fixed for a year. There's a place for being conservative and running old and known stable code, but btrfs is still young and maturing and doesn't match that use-case very well. Either you need stable and mature and an equally stable and mature filesystem is a better fit, at least for a couple more years, or you want newer technology/code and running year-old stuff is behind the curve. Either way, running year-old btrfs code, both kernel and userspace, doesn't make a lot of sense. Perhaps in a couple years it will, after btrfs is more mature and stable and year-old code doesn't mean year-stale bugs, but not currently.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 7 lines · Christian Robottom Reis
# btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch ‘x’: Disk quota exceededThe strange thing is that it doesn't seem to be actually out of space:
# btrfs qgroup show -p -r -e /var | grep 261 0/261 1111810048 391114752 2097152000 0 ---
Replying to myself as I had not yet been subscribed in time to receive a reply; I just upgraded to 3.18.1 and am seeing the same issue on the same subvolume (and on no others).
root@riff:/etc# uname -a
Linux riff 3.18.1-031801-generic #201412170637 SMP Wed Dec 17 11:38:50
UTC 2014 x86_64 x86_64 x86_64 GNU/LinuxIt's quite odd that this specific subvolume acts up, given that there are quite a few others that are closer to the quota:
subvol group total unshared
-------------------------------------------------------------
(unknown) 0/5 1.37G / none 1.16G / none
lxc-template1/rootfs 0/259 0.68G / none 0.10G / 2.00G
machine-2/rootfs 0/261 1.07G / none 0.40G / 2.00G
machine-3/rootfs 0/265 1.17G / none 0.41G / 2.00G
lxc-template2/rootfs 0/271 0.77G / none 0.31G / 2.00G
lxc-template3/rootfs 0/274 0.46G / none 0.02G / 2.00G
machine-4/rootfs 0/283 7.12G / none 6.21G / 10.00G
machine-5/rootfs 0/288 1.05G / none 0.34G / 2.00G
machine-6/rootfs 0/289 11.33G / none 10.74G / 15.00G
machine-7/rootfs 0/290 1.30G / none 0.68G / 2.00G
machine-8/rootfs 0/292 1.00G / none 0.33G / 2.00G
machine-9/rootfs 0/293 1.17G / none 0.38G / 2.00G
machine-10/rootfs 0/306 1.34G / none 0.62G / 2.00G
machine-11/rootfs 0/318 9.49G / none 8.75G / 15.00G
lxc-template4/rootfs 0/320 0.79G / none 0.78G / 2.00G
machine-14/rootfs 0/323 1.10G / none 0.45G / 2.00GThe LWN article suggests that btrfs is quite conservative with quotas, but shouldn't 265, 290, 306, 320 and 323 all be out of quota as well? Or is there a lot else that goes into the calculation beyond the numbers reported by btrfs qgroup show?
What could I do to help investigate further?
Signature · 2 lines
Christian Robottom Reis | [+55 16] 3376 0125 | http://async.com.br/~kiko CEO, Async Open Source | [+55 16] 9 9112 6430 | http://launchpad.net/~kiko
Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as excerpted:
Quoted 12 lines · Christian Robottom Reis
Quoted 7 lines
# btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch ‘x’: Disk quota exceededThe strange thing is that it doesn't seem to be actually out of space:
# btrfs qgroup show -p -r -e /var | grep 261 0/261 1111810048 391114752 2097152000 0 ---Replying to myself as I had not yet been subscribed in time to receive a reply; I just upgraded to 3.18.1 and am seeing the same issue on the same subvolume (and on no others).
Looking at the thread here on gmane.org (list2news and list2web gateway), it appears my reply was the only reply in any case, and it was general as I don't run quotas myself.
Basically I suggested upgrading, as the quota code as some rather huge bugs in it (quotas could go seriously negative!) with the old versions you were running. But you've upgraded at least the kernel now (userspace you didn't say).
Here's a link to the thread on the gmane web interface for completeness, but the above about covers my reply, as I said the only one until your thread bump and my reply here, so there's not much new there unless someone posts further followups to this thread...
http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Just as a follow-up, I upgraded btrfs-tools and the kernel again. I currently have a filesystem which reports 1G exclusive use:
root@riff# btrfs qg show -r -e /var -p -c qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/261 1.52GiB 1.01GiB 0.00B 100.00GiB --- ---
This filesystem reports over quota, and removing the quota fixes that:
root@riff# touch x touch: cannot touch ‘x’: Disk quota exceeded root@riff# btrfs qg limit -e none 261 /var root@riff# touch x root@riff#
So at the moment quotas are pretty much unusable in kernel 3.18.6/tools 3.18.2, at least for my use case, and that's a bit surprising since there isn't anything very interesting about it (other than it contains a bunch of lxc-cloned rootfs).
I've proactively added Yang who has submitted a few patches on quota checking recently just to let me know if he thinks that this should be fixed with a trunk kernel, or if he'd like to investigate or consider this further. Thanks!
Quoted 43 lines · Duncan
Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as excerpted:
Quoted 12 lines · Christian Robottom Reis
Quoted 7 lines
# btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch ‘x’: Disk quota exceededThe strange thing is that it doesn't seem to be actually out of space:
# btrfs qgroup show -p -r -e /var | grep 261 0/261 1111810048 391114752 2097152000 0 ---Replying to myself as I had not yet been subscribed in time to receive a reply; I just upgraded to 3.18.1 and am seeing the same issue on the same subvolume (and on no others).
Looking at the thread here on gmane.org (list2news and list2web gateway), it appears my reply was the only reply in any case, and it was general as I don't run quotas myself.
Basically I suggested upgrading, as the quota code as some rather huge bugs in it (quotas could go seriously negative!) with the old versions you were running. But you've upgraded at least the kernel now (userspace you didn't say).
Here's a link to the thread on the gmane web interface for completeness, but the above about covers my reply, as I said the only one until your thread bump and my reply here, so there's not much new there unless someone posts further followups to this thread...
http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491
Signature · 8 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Christian Robottom Reis | [+1] 612 888 4935 | http://launchpad.net/~kiko Canonical VP Hyperscale | [+55 16] 9 9112 6430
-------- Original Message --------
Subject: Re: Quota limit question
From: Christian Robottom Reis <[email protected]>
To: Duncan <[email protected]>
Date: 2015年03月07日 05:44
Quoted 7 lines
Just as a follow-up, I upgraded btrfs-tools and the kernel again. I currently have a filesystem which reports 1G exclusive use:
root@riff# btrfs qg show -r -e /var -p -c qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/261 1.52GiB 1.01GiB 0.00B 100.00GiB --- ---
It's recommended to sync the btrfs before 'qg show' command, since quota tree is not updated until commit transaction.
Quoted 10 lines
This filesystem reports over quota, and removing the quota fixes that:
root@riff# touch x touch: cannot touch ‘x’: Disk quota exceeded root@riff# btrfs qg limit -e none 261 /var root@riff# touch x root@riff#
Quoted 4 lines
So at the moment quotas are pretty much unusable in kernel 3.18.6/tools 3.18.2, at least for my use case, and that's a bit surprising since there isn't anything very interesting about it (other than it contains a bunch of lxc-cloned rootfs).
Yes, Btrfs really has some problems, from yours to other annoying incorrect values.
[Some guess]
For your case, I'm afraid your reserved space in your qgroup takes too much space, and it's definitely a bug. Maybe some btrfs_qgroup_reserve() leaking?
[Reproducer]
Would you please describe your workload? Or some easy-to-reproduce scripts?
[Image dump/debug tree]
Another one which may help is your btrfs-debug-tree output on your quota tree. You can dump it by "btrfs-debug-tree -t 8 <DEVICE>".
Or full disk metadata dump by "btrfs-image -c9 <DEVICE>"
Thanks,
Qu
Quoted 50 lines
I've proactively added Yang who has submitted a few patches on quota checking recently just to let me know if he thinks that this should be fixed with a trunk kernel, or if he'd like to investigate or consider this further. Thanks!
Quoted 43 lines · Duncan
Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as excerpted:
Quoted 12 lines · Christian Robottom Reis
Quoted 7 lines
# btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch ‘x’: Disk quota exceededThe strange thing is that it doesn't seem to be actually out of space:
# btrfs qgroup show -p -r -e /var | grep 261 0/261 1111810048 391114752 2097152000 0 ---Replying to myself as I had not yet been subscribed in time to receive a reply; I just upgraded to 3.18.1 and am seeing the same issue on the same subvolume (and on no others).
Looking at the thread here on gmane.org (list2news and list2web gateway), it appears my reply was the only reply in any case, and it was general as I don't run quotas myself.
Basically I suggested upgrading, as the quota code as some rather huge bugs in it (quotas could go seriously negative!) with the old versions you were running. But you've upgraded at least the kernel now (userspace you didn't say).
Here's a link to the thread on the gmane web interface for completeness, but the above about covers my reply, as I said the only one until your thread bump and my reply here, so there's not much new there unless someone posts further followups to this thread...
http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sorry about the late reply!!
Quoted 11 lines · Qu Wenruo
-------- Original Message --------
Subject: Re: Quota limit question
From: Christian Robottom Reis <[email protected]>
To: Duncan <[email protected]>
Date: 2015年03月07日 05:44Quoted 3 lines
Just as a follow-up, I upgraded btrfs-tools and the kernel again. I currently have a filesystem which reports 1G exclusive use:
Quoted 6 lines
Quoted 5 lines
root@riff# btrfs qg show -r -e /var -p -c qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/261 1.52GiB 1.01GiB 0.00B 100.00GiB --- ---It's recommended to sync the btrfs before 'qg show' command, since quota tree is not updated until commit transaction.
Yes, sync is required be fore qg show now. There is no ioctl to get qgroup info from kernel space, so we have to read the data from disk then we have to do a sync before we getting the correct information.
Quoted 10 lines
Quoted 10 lines
This filesystem reports over quota, and removing the quota fixes that:
root@riff# touch x touch: cannot touch ‘x’: Disk quota exceeded root@riff# btrfs qg limit -e none 261 /var root@riff# touch x root@riff#
Quoted 8 lines
Quoted 4 lines
So at the moment quotas are pretty much unusable in kernel 3.18.6/tools 3.18.2, at least for my use case, and that's a bit surprising since there isn't anything very interesting about it (other than it contains a bunch of lxc-cloned rootfs).
Yes, Btrfs really has some problems, from yours to other annoying incorrect values.
[Some guess]
For your case, I'm afraid your reserved space in your qgroup takes too much space, and it's definitely a bug. Maybe somebtrfs_qgroup_reserve()leaking?
Agreed, there must be a reserve leaking in your use case. And I have posted some patches about qgroup.reserved. some of them is about reserved leaking.
Could you try the tree :
btrfs:https://[email protected]/yangdongsheng/linux.git qgroup_type
btrfs-progs:https://[email protected]/yangdongsheng/btrfs-progs.git qgroup_type
It could be working better but I can not guarantee your problem here will be solved by it.
Quoted 8 lines
[Reproducer]
Would you please describe your workload? Or some easy-to-reproduce scripts?[Image dump/debug tree]
Another one which may help is your btrfs-debug-tree output on your quota tree. You can dump it by "btrfs-debug-tree -t 8 <DEVICE>".
Or full disk metadata dump by "btrfs-image -c9 <DEVICE>"
A reproducer is great for investigation. "btrfs-debug-tree" is important to us.
Thanx
Yang
Quoted 56 lines
Thanks,
QuQuoted 51 lines
I've proactively added Yang who has submitted a few patches on quota checking recently just to let me know if he thinks that this should be fixed with a trunk kernel, or if he'd like to investigate or consider this further. Thanks!
Quoted 44 lines · Duncan
Christian Robottom Reis posted on Tue, 23 Dec 2014 18:36:02 -0200 as excerpted:
Quoted 5 lines · Christian Robottom Reis
Quoted 4 lines
# btrfs qgroup limit 2000m 0/261 . && touch x touch: cannot touch ‘x’: Disk quota exceededQuoted 8 lines
Quoted 4 lines
The strange thing is that it doesn't seem to be actually out of space:
# btrfs qgroup show -p -r -e /var | grep 261 0/261 1111810048 391114752 2097152000 0 ---Replying to myself as I had not yet been subscribed in time to receive a reply; I just upgraded to 3.18.1 and am seeing the same issue on the same subvolume (and on no others).
Looking at the thread here on gmane.org (list2news and list2web gateway), it appears my reply was the only reply in any case, and it was general as I don't run quotas myself.
Basically I suggested upgrading, as the quota code as some rather huge bugs in it (quotas could go seriously negative!) with the old versions you were running. But you've upgraded at least the kernel now (userspace you didn't say).
Here's a link to the thread on the gmane web interface for completeness, but the above about covers my reply, as I said the only one until your thread bump and my reply here, so there's not much new there unless someone posts further followups to this thread...
http://comments.gmane.org/gmane.comp.file-systems.btrfs/41491
Signature · 8 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html.