Disk quota exceeded

I've create a subvolume
CMD: btrfs quota enable /export/shared
CMD: btrfs subvol create /export/shared/TV/files
Create subvolume '/export/shared/TV/files'
CMD: btrfs quota rescan /export/shared/TV/files

I've assigned a small quota to it:
CMD: btrfs qgroup limit 100M /export/shared/TV/files

I've also disabled copy on write:
CMD: chattr -R -V +C /export/shared/TV/files chattr 1.42.12 (29-Aug-2014)
Flags of /export/shared/TV/files set as ---------------C

CMD: sync; btrfs qgroup show -r /export/shared/TV/files

qgroupid         rfer         excl     max_rfer
--------         ----         ----     --------

0/5 2.11TiB 2.11TiB none
0/1894 16.00KiB 16.00KiB 100.00MiB

I created one file until I exceeded the quota using:
CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded 50+0 records in
49+0 records out
51380224 bytes (51 MB) copied, 0.0238218 s, 2.2 GB/s

then I removed the file
CMD: rm /export/shared/TV/files/junk1

I tried to create it again (and only got 900k written before the quota was exceeded):
CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded 1+0 records in
0+0 records out
917504 bytes (918 kB) copied, 0.000568984 s, 1.6 GB/s

then I removed the file
CMD: rm /export/shared/TV/files/junk1

I tried to create it again (and this time got 900k again written before the quota was exceeded):
dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
dd: error writing ‘/export/shared/TV/files/junk1’: Disk quota exceeded
1+0 records in
0+0 records out
917504 bytes (918 kB) copied, 0.000943652 s, 972 MB/s

The usage looks ok:
CMD: sync; btrfs qgroup show -r /export/shared/TV/files

qgroupid         rfer         excl     max_rfer
--------         ----         ----     --------

0/5 3.10TiB 3.10TiB none
0/1894 912.00KiB 912.00KiB 100.00MiB

But I can not create any more files:
CMD: dd if=/dev/zero of=/export/shared/TV/files/junk1 count=50 bs=1M
dd: failed to open ‘/export/shared/TV/files/junk1’: Disk quota exceeded

CMD: lsattr /export/shared/TV/files
---------------C /export/shared/TV/files/junk1

During other tests I have managed to get it to the point where I can't delete the file (since disk quota exceeded).
I've waited many minutes to ensure commits etc. The only way I can seemingly consistently get the ability to write to the directory again is to reboot. Immediately after a reboot I can create files as expected.

It seems like copy on write is still functioning for this directory? or quota results have delays? I've tried mounting the subvolume to another location, but same results.

Is this happening because quotas are not yet considered stable? Is there something I am doing wrong?

Other Info:
The fstab entry:
LABEL=SHARED /export/shared btrfs defaults,compress=lzo,relatime,nofail 0 1

uname -a
Linux gw.lambert.rd.to 4.2.7-200.fc22.x86_64 #1 SMP Thu Dec 10 03:28:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

btrfs --version
btrfs-progs v4.2.2

btrfs fi show
Label: 'SHARED' uuid: 199672d4-69ef-4ecf-865a-851bed0167d5
Total devices 5 FS bytes used 3.10TiB
devid 1 size 2.73TiB used 1.27TiB path /dev/sdc
devid 2 size 2.73TiB used 1.27TiB path /dev/sdf
devid 3 size 2.73TiB used 1.27TiB path /dev/sdh
devid 4 size 2.73TiB used 1.27TiB path /dev/sde
devid 5 size 2.73TiB used 1.27TiB path /dev/sdd

btrfs fi df /export/shared
Data, RAID10: total=3.16TiB, used=3.09TiB System, RAID10: total=64.00MiB, used=352.00KiB Metadata, RAID10: total=9.00GiB, used=7.61GiB GlobalReserve, single: total=512.00MiB, used=0.00B

dmesg:
http://edcint.co.nz/tmp/dmesg_20151219.log

Matthew Jurgens posted on Sat, 19 Dec 2015 21:38:50 +1100 as excerpted:

[wrote only ~50 M of files, can't write more]

[Disclaimer: List regular and btrfs user. Not a dev. My own use-case doesn't involve quotas or even subvolumes, so the following is based purely on what I've seen on-list.]

Quotas are definitely not stable, thru 4.3 at least, and you're running 4.2.

IIRC They've on the product of the second major rewrite (so third try) now and there have always been issues. While there's hard work going into fixing the issues, full fixes have remained at least a kernel cycle or two out for quite some time now. AFAIK, 4.4 is supposed to fix some more issues, but I'm not sure if it fixes all known issues yet or if there's more remaining, and even if it fixes all known issues, I'd suggest waiting at least a complete kernel cycle and preferably two with no known issues, before considering quotas actually reasonably dependable, as given the history, there's a fair chance more issues will popup.

Additionally, quotas thru at least 4.3 still have bad scaling issues that slow down maintenance such as balance. (I'm not sure if check is affected or not, but scrub shouldn't be.)

So I'd strongly suggest leaving quotas off thru the 4.4 cycle at least. If 4.4 and the 4.5 development cycle ends up with no further issues, you may wish to consider enabling them for 4.5, tho more conservative users will wait another cycle and see if there's no further issues reported thru the 4.6 development cycle as well, enabling them once they're on it.

(Since 4.4 will be an LTS kernel, if you're sticking to it, you could consider enabling quotas on updates after 4.5 or 4.6 release, if there have been no reports of quota issues on 4.4 by then.)

Meanwhile, a few more quota related factors to consider:

1) As I'm not a quota user I can't say for sure on this, but it seems mighty coincidental that it failed at just over half the set quota... and you're running btrfs raid10 data, so btrfs is keeping two copies and doubling the usage. Maybe it's tracking the actual doubled due to the raid10 usage instead of the single-copy value? (This actually sounds like some of the sort of internal logic bugs they've been fixing tho I'm unaware of one exactly like this, so you might try a current kernel 4.4-rc and see if the results are different, and/or try it on a different btrfs with replication set to single mode.)

2) Btrfs metadata is always copy-on-write. That can't be disabled. Disabling COW only disables it for data.

I remember some discussion of whether quotas should track data only or include metadata as well, or possibly make it optional to track both or set quotas separately for each, but IDR the results of those discussions and as I don't use them myself...

That would explain not being able to delete the file at times, because doing so would change the metadata and thus require COWing it, and if it's quota tracked and reporting over quota errors...

3) While this shouldn't apply to the scenario above because you followed the recommendation and set nocow on the dir and /then/ created the file, so the file should have inherited the nocow, and I suppose you know this already, but for the benefit of others following alone, it should be noted that nocow doesn't necessarily take effect on files that already have content -- they need to be zero-sized for it to take effect.

The recommended procedure to avoid the problem is exactly what you did, set nocow on the dir and take care to either create new files (as you did) or copy/move them into place without reflinking, so they're actually created before they have any content, so the nocow inherited from the dir will actually take effect.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Matthew Jurgens wrote on 2015/12/19 21:38 +1100:

The excl/rfer numbers will always be accurate, as that what 4.2 quota rework does.

That's what exactly I fixed in 4.3-rc1.
And the operation to trigger the bug is almost the same with xfstest test case: btrfs/099.

Btrfs doesn't handle quota reserved space well, and caused reserved space leak. (fixed in 4.3-rc1, but more test would be better)

As long as you did overwrite, no matter whether you enabled or disabled CoW, it will leak reserved space, and the reserved space won't be freed.

Finally eating up your quota space.

You can either try latest 4.3-rc, which does not only include the quota fix, but also a lot of delayed ref fixes.

Or, there is a workaround:
Umount the fs and mount it again.
This will free the leaked space and make it work again for a short time, until your next overwrite.

Thanks,
Qu