"Quota Ignored On write" problem still exist with 3.16-rc5
Hi Josef, Chris,
I found "Quota Ignored On write" problem still exist with 3.16-rc5, which Kevin reported before.
Kevin's report:
https://www.mail-archive.com/[email protected]/msg35292.html
The result of bisect:
https://www.mail-archive.com/[email protected]/msg35304.html
Quoted 11 lines
I bisected and found the bad commit is the following patch.
===============================================================================
commitfcebe4562dec83b3f8d3088d77584727b09130b2Author: Josef Bacik <[email protected]> Date: Tue May 13 17:30:47 2014 -0700Btrfs: rework qgroup accounting
===============================================================================Josef, please take a look at this patch.
Reproducer:
https://www.mail-archive.com/[email protected]/msg35299.html
Could you tell me the progress of fixing this bug? In addition, could you fix it by 3.16?
command log:
===============================================================================
# ./test.sh + uname -a Linux luna.soft.fujitsu.com 3.16.0-rc5 #2 SMP Tue Jul 15 13:39:46 JST 2014 x86_64 x86_64 x86_64 GNU/Linux + df -T /test7 Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdc7 btrfs 29296640 1536 27169536 1% /test7 + btrfs quota ena /test7 + cd /test7 + btrfs sub cre test Create subvolume './test' + btrfs sub l -a /test7 ID 270 gen 66 top level 5 path test + btrfs qg lim 1G test # limit test subvol to 1GB + btrfs qg show -pcre /test7 qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/5 16384 16384 0 0 --- --- 0/270 16384 16384 1073741824 0 --- --- + dd if=/dev/zero of=test/file0 bs=1M count=2000 2000+0 records in 2000+0 records out 2097152000 bytes (2.1 GB) copied, 9.67876 s, 217 MB/s # write 2GB. It's a bug! + sync + ls -lisaR /test7 /test7: total 20 256 16 drwxr-xr-x 1 root root 8 Jul 18 15:12 . 2 4 drwxr-xr-x. 43 root root 4096 Jul 16 08:34 .. 256 0 drwxr-xr-x 1 root root 10 Jul 18 15:17 test
/test7/test:
total 2048016
256 0 drwxr-xr-x 1 root root 10 Jul 18 15:17 .
256 16 drwxr-xr-x 1 root root 8 Jul 18 15:12 ..
257 2048000 -rw-r--r-- 1 root root 2097152000 Jul 18 15:17 file0
+ btrfs qg show -pcre /test7
qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/5 16384 16384 0 0 --- --- 0/270 2097168384 2097168384 1073741824 0 --- ---
+ btrfs quota dis /test7
+ btrfs sub del test
Transaction commit: none (default)
Delete subvolume '/test7/test'
+ set +x
#
===============================================================================
NOTE:
The reproducer here (./test.sh) is a bit different from above-mentioned one because of some reason.
Thanks,
Satoru
Quoted 10 lines · Satoru Takeuchi
Hi Josef, Chris,
I found "Quota Ignored On write" problem still exist with 3.16-rc5, which Kevin reported before.
Kevin's report:
https://www.mail-archive.com/[email protected]/msg35292.htmlThe result of bisect:
https://www.mail-archive.com/[email protected]/msg35304.html
I guess this is because Josef's patch delayed qgroup accounting, it will cause @refer and @excl updating very late...
The patch maybe optimize to merge some delayed refs(for example), but it updates qgroup accounting when commiting transaction which will be very late,
we may have accumulated many data..
Thanks,
Wang
Quoted 81 lines
Quoted 5 lines
I bisected and found the bad commit is the following patch.
===============================================================================
commitfcebe4562dec83b3f8d3088d77584727b09130b2Quoted 7 lines
Author: Josef Bacik <[email protected]> Date: Tue May 13 17:30:47 2014 -0700Btrfs: rework qgroup accounting
===============================================================================Josef, please take a look at this patch.
Reproducer:
https://www.mail-archive.com/[email protected]/msg35299.htmlCould you tell me the progress of fixing this bug? In addition, could you fix it by 3.16?
command log:
===============================================================================# ./test.sh + uname -a Linux luna.soft.fujitsu.com 3.16.0-rc5 #2 SMP Tue Jul 15 13:39:46 JST 2014 x86_64 x86_64 x86_64 GNU/Linux + df -T /test7 Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdc7 btrfs 29296640 1536 27169536 1% /test7 + btrfs quota ena /test7 + cd /test7 + btrfs sub cre test Create subvolume './test' + btrfs sub l -a /test7 ID 270 gen 66 top level 5 path test + btrfs qg lim 1G test # limit test subvol to 1GB + btrfs qg show -pcre /test7 qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/5 16384 16384 0 0 --- --- 0/270 16384 16384 1073741824 0 --- --- + dd if=/dev/zero of=test/file0 bs=1M count=2000 2000+0 records in 2000+0 records out 2097152000 bytes (2.1 GB) copied, 9.67876 s, 217 MB/s # write 2GB. It's a bug! + sync + ls -lisaR /test7 /test7: total 20 256 16 drwxr-xr-x 1 root root 8 Jul 18 15:12 . 2 4 drwxr-xr-x. 43 root root 4096 Jul 16 08:34 .. 256 0 drwxr-xr-x 1 root root 10 Jul 18 15:17 test/test7/test:
total 2048016
256 0 drwxr-xr-x 1 root root 10 Jul 18 15:17 .
256 16 drwxr-xr-x 1 root root 8 Jul 18 15:12 ..
257 2048000 -rw-r--r-- 1 root root 2097152000 Jul 18 15:17 file0
+ btrfs qg show -pcre /test7qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/5 16384 16384 0 0 --- --- 0/270 2097168384 2097168384 1073741824 0 --- ---+ btrfs quota dis /test7
+ btrfs sub del test
Transaction commit: none (default)
Delete subvolume '/test7/test'
+ set +x
#
===============================================================================NOTE:
The reproducer here (./test.sh) is a bit different from above-mentioned one because of some reason.Thanks,
Satoru--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
.
Hi Wang,
Quoted 11 lines · Wang Shilong
Quoted 10 lines · Satoru Takeuchi
Hi Josef, Chris,
I found "Quota Ignored On write" problem still exist with 3.16-rc5, which Kevin reported before.
Kevin's report:
https://www.mail-archive.com/[email protected]/msg35292.htmlThe result of bisect:
https://www.mail-archive.com/[email protected]/msg35304.html
Quoted 7 lines
I guess this is because Josef's patch delayed qgroup accounting, it will cause @refer and @excl updating very late...
The patch maybe optimize to merge some delayed refs(for example), but it updates qgroup accounting when commiting transaction which will be very late,
we may have accumulated many data..
Thank you for your comment. I know of the code logic which caused this problem. However, what I want to say here is that this problem should be fixed as soon as possible. It is a important regression problem and we've already know the root cause.
If it's impossible to fix it by releasing 3.16, I consider this patch should be reverted.
Thanks,
Satoru
Quoted 90 lines
Thanks,
WangQuoted 72 lines
Quoted 5 lines
I bisected and found the bad commit is the following patch.
===============================================================================
commitfcebe4562dec83b3f8d3088d77584727b09130b2Quoted 7 lines
Author: Josef Bacik <[email protected]> Date: Tue May 13 17:30:47 2014 -0700Btrfs: rework qgroup accounting
===============================================================================Josef, please take a look at this patch.
Reproducer:
https://www.mail-archive.com/[email protected]/msg35299.htmlCould you tell me the progress of fixing this bug? In addition, could you fix it by 3.16?
command log:
===============================================================================# ./test.sh + uname -a Linux luna.soft.fujitsu.com 3.16.0-rc5 #2 SMP Tue Jul 15 13:39:46 JST 2014 x86_64 x86_64 x86_64 GNU/Linux + df -T /test7 Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdc7 btrfs 29296640 1536 27169536 1% /test7 + btrfs quota ena /test7 + cd /test7 + btrfs sub cre test Create subvolume './test' + btrfs sub l -a /test7 ID 270 gen 66 top level 5 path test + btrfs qg lim 1G test # limit test subvol to 1GB + btrfs qg show -pcre /test7 qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/5 16384 16384 0 0 --- --- 0/270 16384 16384 1073741824 0 --- --- + dd if=/dev/zero of=test/file0 bs=1M count=2000 2000+0 records in 2000+0 records out 2097152000 bytes (2.1 GB) copied, 9.67876 s, 217 MB/s # write 2GB. It's a bug! + sync + ls -lisaR /test7 /test7: total 20 256 16 drwxr-xr-x 1 root root 8 Jul 18 15:12 . 2 4 drwxr-xr-x. 43 root root 4096 Jul 16 08:34 .. 256 0 drwxr-xr-x 1 root root 10 Jul 18 15:17 test/test7/test:
total 2048016
256 0 drwxr-xr-x 1 root root 10 Jul 18 15:17 .
256 16 drwxr-xr-x 1 root root 8 Jul 18 15:12 ..
257 2048000 -rw-r--r-- 1 root root 2097152000 Jul 18 15:17 file0
+ btrfs qg show -pcre /test7qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/5 16384 16384 0 0 --- --- 0/270 2097168384 2097168384 1073741824 0 --- ---+ btrfs quota dis /test7
+ btrfs sub del test
Transaction commit: none (default)
Delete subvolume '/test7/test'
+ set +x
#
===============================================================================NOTE:
The reproducer here (./test.sh) is a bit different from above-mentioned one because of some reason.Quoted 9 lines
Thanks,
Satoru--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
.--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Satoru-san,
Quoted 21 lines · Satoru Takeuchi
Hi Wang,
Quoted 11 lines · Wang Shilong
Quoted 10 lines · Satoru Takeuchi
Hi Josef, Chris,
I found "Quota Ignored On write" problem still exist with 3.16-rc5, which Kevin reported before.
Kevin's report:
https://www.mail-archive.com/[email protected]/msg35292.htmlThe result of bisect:
https://www.mail-archive.com/[email protected]/msg35304.htmlQuoted 7 lines
I guess this is because Josef's patch delayed qgroup accounting, it will cause @refer and @excl updating very late...
The patch maybe optimize to merge some delayed refs(for example), but it updates qgroup accounting when commiting transaction which will be very late,
we may have accumulated many data..
Quoted 7 lines
Thank you for your comment. I know of the code logic which caused this problem. However, what I want to say here is that this problem should be fixed as soon as possible. It is a important regression problem and we've already know the root cause.
If it's impossible to fix it by releasing 3.16, I consider this patch should be reverted.
Since Btrfs Quota function is under heavy development, and should be considered as *broken*.
I think we'd better close Btrfs quota function(Like snapshot-aware function) until we really sit down and solve everything.
Thanks,
Wang
Quoted 93 lines
Thanks,
SatoruQuoted 82 lines
Thanks,
WangQuoted 71 lines
Quoted 11 lines
I bisected and found the bad commit is the following patch.
===============================================================================
commitfcebe4562dec83b3f8d3088d77584727b09130b2Author: Josef Bacik <[email protected]> Date: Tue May 13 17:30:47 2014 -0700Btrfs: rework qgroup accounting
===============================================================================Josef, please take a look at this patch.
Reproducer:
https://www.mail-archive.com/[email protected]/msg35299.htmlCould you tell me the progress of fixing this bug? In addition, could you fix it by 3.16?
command log:
===============================================================================# ./test.sh + uname -a Linux luna.soft.fujitsu.com 3.16.0-rc5 #2 SMP Tue Jul 15 13:39:46 JST 2014 x86_64 x86_64 x86_64 GNU/Linux + df -T /test7 Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/sdc7 btrfs 29296640 1536 27169536 1% /test7 + btrfs quota ena /test7 + cd /test7 + btrfs sub cre test Create subvolume './test' + btrfs sub l -a /test7 ID 270 gen 66 top level 5 path test + btrfs qg lim 1G test # limit test subvol to 1GB + btrfs qg show -pcre /test7 qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/5 16384 16384 0 0 --- --- 0/270 16384 16384 1073741824 0 --- --- + dd if=/dev/zero of=test/file0 bs=1M count=2000 2000+0 records in 2000+0 records out 2097152000 bytes (2.1 GB) copied, 9.67876 s, 217 MB/s # write 2GB. It's a bug! + sync + ls -lisaR /test7 /test7: total 20 256 16 drwxr-xr-x 1 root root 8 Jul 18 15:12 . 2 4 drwxr-xr-x. 43 root root 4096 Jul 16 08:34 .. 256 0 drwxr-xr-x 1 root root 10 Jul 18 15:17 test/test7/test:
total 2048016
256 0 drwxr-xr-x 1 root root 10 Jul 18 15:17 .
256 16 drwxr-xr-x 1 root root 8 Jul 18 15:12 ..
257 2048000 -rw-r--r-- 1 root root 2097152000 Jul 18 15:17 file0
+ btrfs qg show -pcre /test7qgroupid rfer excl max_rfer max_excl parent child -------- ---- ---- -------- -------- ------ ----- 0/5 16384 16384 0 0 --- --- 0/270 2097168384 2097168384 1073741824 0 --- ---+ btrfs quota dis /test7
+ btrfs sub del test
Transaction commit: none (default)
Delete subvolume '/test7/test'
+ set +x
#
===============================================================================NOTE:
The reproducer here (./test.sh) is a bit different from above-mentioned one because of some reason.Quoted 9 lines
Thanks,
Satoru--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
.Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html.
Quoted 22 lines · Satoru Takeuchi
Hi Wang,
Quoted 11 lines · Wang Shilong
Quoted 10 lines · Satoru Takeuchi
Hi Josef, Chris,
I found "Quota Ignored On write" problem still exist with 3.16-rc5, which Kevin reported before.
Quoted 7 lines
I guess this is because Josef's patch delayed qgroup accounting, it will cause @refer and @excl updating very late...
The patch maybe optimize to merge some delayed refs(for example), but it updates qgroup accounting when commiting transaction which will be very late,
we may have accumulated many data..
Quoted 8 lines
Thank you for your comment. I know of the code logic which caused this problem. However, what I want to say here is that this problem should be fixed as soon as possible. It is a important regression problem and we've already know the root cause.
If it's impossible to fix it by releasing 3.16, I consider this patch should be reverted.
That's not happening. Qgroups have always been broken in one way or another, now with my stuff the accounting is always correct and we don't have a huge performance impact with or without qgroups. We'll fix it but there's no way we're reverting this work. Thanks,
Josef
Quoted 30 lines · Josef Bacik
Quoted 21 lines · Satoru Takeuchi
Hi Wang,
Quoted 11 lines · Wang Shilong
Quoted 10 lines · Satoru Takeuchi
Hi Josef, Chris,
I found "Quota Ignored On write" problem still exist with 3.16-rc5, which Kevin reported before.
Quoted 7 lines
I guess this is because Josef's patch delayed qgroup accounting, it will cause @refer and @excl updating very late...
The patch maybe optimize to merge some delayed refs(for example), but it updates qgroup accounting when commiting transaction which will be very late,
we may have accumulated many data..Quoted 8 lines
Thank you for your comment. I know of the code logic which caused this problem. However, what I want to say here is that this problem should be fixed as soon as possible. It is a important regression problem and we've already know the root cause.
If it's impossible to fix it by releasing 3.16, I consider this patch should be reverted.
Quoted 7 lines
That's not happening. Qgroups have always been broken in one way or another, now with my stuff the accounting is always correct and we don't have a huge performance impact with or without qgroups. We'll fix it but there's no way we're reverting this work. Thanks,
Josef
How/when does the accounting happen? Obviously we want the accounting to be correct, but the overrun seems to indicate that the accounting isn't happening early enough. From briefly looking at the patches, it looks to be that the intent is to account when a new extent is allocated, but it seems that this isn't happening if its still allowed to allocate post quota, or is the accounting postponed till after a specified time, etc?
-Kevin