A question for kernel code in btrfs_run_qgroups()
Hi, I'm reading btrfs code for contributing. I have a question about code in btrfs_run_qgroups().
It seems that it updates dirty qgroups modified in transaction. And it iterates dirty qgroups with locking and unlocking qgroup_lock for protecting dirty_qgroups list. According to code, It locks before enter the loop and pick a entry and unlock. At the end of loop, It locks again for next loop. And after loop, it set some flags and unlock.
I wonder that it should be locked with setting STATUS_FLAG_ON? if not, is there unnecessary locking in last loop?
Quoted 4 lines · Sidong Yang
Hi, I'm reading btrfs code for contributing. I have a question about code in
btrfs_run_qgroups().It seems that it updates dirty qgroups modified in transaction.
Yep.
Quoted 7 lines
And it iterates dirty qgroups with locking and unlocking
qgroup_lockfor protectingdirty_qgroupslist. According to code, It locks before enter the loop and pick a entry and unlock. At the end of loop, It locks again for next loop. And after loop, it set some flags and unlock.I wonder that it should be locked with setting
STATUS_FLAG_ON? if not, is there unnecessary locking in last loop?
From a quick look, it indeed looks like we don't need to hole qgroup_lock to modify qgroup_flags.
In fact, just lines below, we update qgroup_flags without any lock for INCONSISTENT bit.
Unfortunately we indeed have inconsistent locking for qgroups_flags.
So it's completely possible that we may not need to do modify the qgroup_flags under qgroup_lock.
In fact there are tons of call sites that we don't hold locks for qgroups_flags update.
So if you're interested in contributing to btrfs, starting from sorting the qgroup_lock usage would be a excellent start.
Thanks,
Qu
On Tue, Feb 01, 2022 at 08:21:22AM +0800, Qu Wenruo wrote:
Hi Qu,
Thanks for answering
Quoted 15 lines · Sidong Yang
Quoted 4 lines
Hi, I'm reading btrfs code for contributing. I have a question about code in
btrfs_run_qgroups().It seems that it updates dirty qgroups modified in transaction.
Yep.
Quoted 5 lines
And it iterates dirty qgroups with locking and unlocking
qgroup_lockfor protectingdirty_qgroupslist. According to code, It locks before enter the loop and pick a entry and unlock. At the end of loop, It locks again for next loop. And after loop, it set some flags and unlock.
Quoted 7 lines
Quoted 2 lines
I wonder that it should be locked with setting
STATUS_FLAG_ON? if not, is there unnecessary locking in last loop?From a quick look, it indeed looks like we don't need to hole
qgroup_lockto modifyqgroup_flags.
In fact, just lines below, we updateqgroup_flagswithout any lock for INCONSISTENT bit.
Apparently it is, but I don't know surely that it really don't need to hold lock while modifying qgroup_flags.
It has FLAG_ON that it indicates that quota turned on. I think it should be modified carefully. Or it can be protected by other locks like qgroup_lock or ioctl_lock?
Quoted 3 lines
Unfortunately we indeed have inconsistent locking for
qgroups_flags.
I agree. there is a lot of codes that modify qgroup_flags without lock or with lock.
Quoted 9 lines
So it's completely possible that we may not need to do modify the
qgroup_flagsunderqgroup_lock.In fact there are tons of call sites that we don't hold locks for
qgroups_flagsupdate.So if you're interested in contributing to btrfs, starting from sorting the
qgroup_lockusage would be a excellent start.
Yeah, I really interested in this. but I don't know good way to handle this problem. is it really good way to remove the code holding lock while modifying flags?
Thanks,
Sidong
Quoted 3 lines
Thanks,
Qu
Quoted 29 lines · Sidong Yang
On Tue, Feb 01, 2022 at 08:21:22AM +0800, Qu Wenruo wrote:
Hi Qu,
Thanks for answering
Quoted 15 lines · Sidong Yang
Quoted 4 lines
Hi, I'm reading btrfs code for contributing. I have a question about code in
btrfs_run_qgroups().It seems that it updates dirty qgroups modified in transaction.
Yep.
Quoted 5 lines
And it iterates dirty qgroups with locking and unlocking
qgroup_lockfor protectingdirty_qgroupslist. According to code, It locks before enter the loop and pick a entry and unlock. At the end of loop, It locks again for next loop. And after loop, it set some flags and unlock.Quoted 7 lines
Quoted 2 lines
I wonder that it should be locked with setting
STATUS_FLAG_ON? if not, is there unnecessary locking in last loop?From a quick look, it indeed looks like we don't need to hole
qgroup_lockto modifyqgroup_flags.
In fact, just lines below, we updateqgroup_flagswithout any lock for INCONSISTENT bit.
Quoted 5 lines
Apparently it is, but I don't know surely that it really don't need to hold lock while modifying
qgroup_flags.
It hasFLAG_ONthat it indicates that quota turned on. I think it should be modified carefully. Or it can be protected by other locks likeqgroup_lockorioctl_lock?
In fact, for qgroup_flags, FLAG_ON is rarely changed. Only qgroup enable/disable would change that.
And qgroup enable/disable have to acquire qgroup_ioctl_lock, thus AFAIK FLAG_ON is less a concern.
To me, the concern is around INCONSISTENT flag.
As it can happen at almost any time.
As btrfs_qgroup_trace_extent()/trace_extent_post() can fail and cause qgroup to be inconsistent.
Another location is in the future to make snapshot creation/snapshot drop to mark qgroup INCONSISTENT to avoid super expensive subtree rescan.
Quoted 7 lines
Quoted 4 lines
Unfortunately we indeed have inconsistent locking for
qgroups_flags.I agree. there is a lot of codes that modify
qgroup_flagswithout lock or with lock.
So far we use it just as bit operations, thus it may be convert to set/clear/test_bit() to be atomic, and then get rid of the chaos of locks.
Quoted 8 lines
Quoted 8 lines
So it's completely possible that we may not need to do modify the
qgroup_flagsunderqgroup_lock.In fact there are tons of call sites that we don't hold locks for
qgroups_flagsupdate.
Quoted 6 lines
Quoted 2 lines
So if you're interested in contributing to btrfs, starting from sorting the
qgroup_lockusage would be a excellent start.Yeah, I really interested in this. but I don't know good way to handle this problem. is it really good way to remove the code holding lock while modifying flags?
Maybe you can start by converting it to bit ops and move all the flags operation out of critical sections?
Thanks,
Qu
Quoted 7 lines
Thanks,
SidongQuoted 3 lines
Thanks,
Qu
Quoted 44 lines · Qu Wenruo
Quoted 29 lines · Sidong Yang
On Tue, Feb 01, 2022 at 08:21:22AM +0800, Qu Wenruo wrote:
Hi Qu,
Thanks for answering
Quoted 15 lines · Sidong Yang
Quoted 4 lines
Hi, I'm reading btrfs code for contributing. I have a question about code in
btrfs_run_qgroups().It seems that it updates dirty qgroups modified in transaction.
Yep.
Quoted 5 lines
And it iterates dirty qgroups with locking and unlocking
qgroup_lockfor protectingdirty_qgroupslist. According to code, It locks before enter the loop and pick a entry and unlock. At the end of loop, It locks again for next loop. And after loop, it set some flags and unlock.Quoted 7 lines
Quoted 2 lines
I wonder that it should be locked with setting
STATUS_FLAG_ON? if not, is there unnecessary locking in last loop?From a quick look, it indeed looks like we don't need to hole
qgroup_lockto modifyqgroup_flags.
In fact, just lines below, we updateqgroup_flagswithout any lock for INCONSISTENT bit.Quoted 5 lines
Apparently it is, but I don't know surely that it really don't need to hold lock while modifying
qgroup_flags.
It hasFLAG_ONthat it indicates that quota turned on. I think it should be modified carefully. Or it can be protected by other locks likeqgroup_lockorioctl_lock?In fact, for
qgroup_flags,FLAG_ONis rarely changed. Only qgroup enable/disable would change that.And qgroup enable/disable have to acquire
qgroup_ioctl_lock, thus AFAIKFLAG_ONis less a concern.
Quoted 8 lines
To me, the concern is around INCONSISTENT flag.
As it can happen at almost any time.
As btrfs_qgroup_trace_extent()/trace_extent_post() can fail and cause qgroup to be inconsistent.
Another location is in the future to make snapshot creation/snapshot drop to mark qgroup INCONSISTENT to avoid super expensive subtree rescan.
I agree. INCONSISTENT flag could be set anywhere. And it could make more bigger problem.
Quoted 23 lines
Quoted 8 lines
Quoted 5 lines
Unfortunately we indeed have inconsistent locking for
qgroups_flags.I agree. there is a lot of codes that modify
qgroup_flagswithout lock or with lock.So far we use it just as bit operations, thus it may be convert to set/clear/test_bit() to be atomic, and then get rid of the chaos of locks.
Quoted 11 lines
Quoted 10 lines
So it's completely possible that we may not need to do modify the
qgroup_flagsunderqgroup_lock.In fact there are tons of call sites that we don't hold locks for
qgroups_flagsupdate.So if you're interested in contributing to btrfs, starting from sorting the
qgroup_lockusage would be a excellent start.
Quoted 6 lines
Quoted 3 lines
Yeah, I really interested in this. but I don't know good way to handle this problem. is it really good way to remove the code holding lock while modifying flags?
Maybe you can start by converting it to bit ops and move all the flags operation out of critical sections?
Thanks. I would write a patch that converts the code to bitops first.
Thanks,
Sidong
Quoted 11 lines
Thanks,
QuQuoted 7 lines
Thanks,
SidongQuoted 3 lines
Thanks,
Qu
Quoted 79 lines · Sidong Yang
Quoted 44 lines · Qu Wenruo
Quoted 29 lines · Sidong Yang
On Tue, Feb 01, 2022 at 08:21:22AM +0800, Qu Wenruo wrote:
Hi Qu,
Thanks for answering
Quoted 15 lines · Sidong Yang
Quoted 4 lines
Hi, I'm reading btrfs code for contributing. I have a question about code in
btrfs_run_qgroups().It seems that it updates dirty qgroups modified in transaction.
Yep.
Quoted 5 lines
And it iterates dirty qgroups with locking and unlocking
qgroup_lockfor protectingdirty_qgroupslist. According to code, It locks before enter the loop and pick a entry and unlock. At the end of loop, It locks again for next loop. And after loop, it set some flags and unlock.Quoted 7 lines
Quoted 2 lines
I wonder that it should be locked with setting
STATUS_FLAG_ON? if not, is there unnecessary locking in last loop?From a quick look, it indeed looks like we don't need to hole
qgroup_lockto modifyqgroup_flags.
In fact, just lines below, we updateqgroup_flagswithout any lock for INCONSISTENT bit.Quoted 5 lines
Apparently it is, but I don't know surely that it really don't need to hold lock while modifying
qgroup_flags.
It hasFLAG_ONthat it indicates that quota turned on. I think it should be modified carefully. Or it can be protected by other locks likeqgroup_lockorioctl_lock?In fact, for
qgroup_flags,FLAG_ONis rarely changed. Only qgroup enable/disable would change that.And qgroup enable/disable have to acquire
qgroup_ioctl_lock, thus AFAIKFLAG_ONis less a concern.Quoted 8 lines
To me, the concern is around INCONSISTENT flag.
As it can happen at almost any time.
As btrfs_qgroup_trace_extent()/trace_extent_post() can fail and cause qgroup to be inconsistent.
Another location is in the future to make snapshot creation/snapshot drop to mark qgroup INCONSISTENT to avoid super expensive subtree rescan.I agree. INCONSISTENT flag could be set anywhere. And it could make more bigger problem.
Quoted 23 lines
Quoted 8 lines
Quoted 5 lines
Unfortunately we indeed have inconsistent locking for
qgroups_flags.I agree. there is a lot of codes that modify
qgroup_flagswithout lock or with lock.So far we use it just as bit operations, thus it may be convert to set/clear/test_bit() to be atomic, and then get rid of the chaos of locks.
Quoted 11 lines
Quoted 10 lines
So it's completely possible that we may not need to do modify the
qgroup_flagsunderqgroup_lock.In fact there are tons of call sites that we don't hold locks for
qgroups_flagsupdate.So if you're interested in contributing to btrfs, starting from sorting the
qgroup_lockusage would be a excellent start.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Yeah, I really interested in this. but I don't know good way to handle this problem. is it really good way to remove the code holding lock while modifying flags?
Maybe you can start by converting it to bit ops and move all the flags operation out of critical sections?
Thanks. I would write a patch that converts the code to bitops first.
One thing to mention is, if you're converting the flags operations to bit ops, then please considering protecting it under qgroup_lock.
For now, it's fine as under most cases we just set/clear/test one bit once.
But for future patches, like this one:
https://patchwork.kernel.org/project/linux-btrfs/patch/[email protected]/
Which will introduce a new flag beyond the existing three flags.
And we need to set that flag along with the existing INCONSITENT flag, thus it's no longer atomic, and may need to use spin lock to protect them.
Thanks,
Qu
Quoted 14 lines
Thanks,
SidongQuoted 11 lines
Thanks,
QuQuoted 7 lines
Thanks,
SidongQuoted 3 lines
Thanks,
Qu