blktrace: Avoid sparse warnings when assigning q->blk_trace
Subsystems (2)
PATCH 1/1 blktrace: Avoid sparse warnings when assigning q->blk_trace
Mostly for historical reasons, q->blk_trace is assigned through xchg() and cmpxchg() atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes to q->blk_trace happen under q->blk_trace_mutex. So let's just replace xchg() with rcu_replace_pointer() and cmpxchg() with explicit check and rcu_assign_pointer(). This makes the code more efficient and sparse happy.
--- kernel/trace/blktrace.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
kernel/trace/blktrace.c +10 -10
@@ -344,7 +344,8 @@ static int __blk_trace_remove(struct request_queue *q) {struct blk_trace *bt; - bt =xchg (&q->blk_trace,NULL ); + bt =rcu_replace_pointer (q->blk_trace,NULL , +lockdep_is_held (&q->blk_trace_mutex));if (!bt)return -EINVAL ;
@@ -485,6 +486,10 @@ static int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,if (!blk_debugfs_root)return -ENOENT ; +if (rcu_dereference_protected (q->blk_trace, +lockdep_is_held (&q->blk_trace_mutex))) +return -EBUSY ; +strncpy (buts->name, name,BLKTRACE_BDEV_SIZE ); buts->name[BLKTRACE_BDEV_SIZE -1 ] ='\0' ;
@@ -543,10 +548,7 @@ static int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev, bt->pid = buts->pid; bt->trace_state = Blktrace_setup; - ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto err; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref (); ret =0 ;
@@ -1637,7 +1639,8 @@ static int blk_trace_remove_queue(struct request_queue *q) {struct blk_trace *bt; - bt =xchg (&q->blk_trace,NULL ); + bt =rcu_replace_pointer (q->blk_trace,NULL , +lockdep_is_held (&q->blk_trace_mutex));if (bt ==NULL )return -EINVAL ;
@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q,blk_trace_setup_lba (bt, bdev); - ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto free_bt; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref ();return 0 ;
Signature · 1 line
2.16.4
(+Luis)
Quoted 11 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>
How about adding a reference to commit c780e86dd48e ("blktrace: Protect q->blk_trace with RCU") in the description of this patch?
kernel/trace/blktrace.c static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 11 lines
kernel/trace/blktrace.c +1
-4@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q,blk_trace_setup_lba (bt, bdev); - ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto free_bt; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref ();return 0 ;
This changes a conditional assignment of q->blk_trace into an unconditional assignment. Shouldn't q->blk_trace only be assigned if q->blk_trace == NULL?
Thanks,
Bart.
Quoted 4 lines · Bart Van Assche
Quoted 4 lines
kernel/trace/blktrace.c +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q,blk_trace_setup_lba (bt, bdev);
Quoted 10 lines
kernel/trace/blktrace.c static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
kernel/trace/blktrace.c +1
-4- ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto free_bt; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref ();return 0 ;Shouldn't
q->blk_traceonly be assigned ifq->blk_trace== NULL?
Yes but the old call checked for that and left it as NULL if it was not NULL.
I have a patch in my series which checks for q->blkt_trace *early* prior to continuing to avoid concurrent calls proactively, otherwise this will fail only at the very end.
Luis
Quoted 12 lines · Bart Van Assche
(+Luis)
Quoted 9 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.
Quoted 5 lines
Quoted 2 lines
Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>How about adding a reference to commit
c780e86dd48e("blktrace: Protectq->blk_tracewith RCU") in the description of this patch?
Yes, that's probably a good idea.
Quoted 4 lines
Quoted 4 lines
kernel/trace/blktrace.c +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q,blk_trace_setup_lba (bt, bdev);
Quoted 11 lines
kernel/trace/blktrace.c static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
kernel/trace/blktrace.c +1
-4- ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto free_bt; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref ();return 0 ;This changes a conditional assignment of
q->blk_traceinto an unconditional assignment. Shouldn'tq->blk_traceonly be assigned ifq->blk_trace== NULL?
Yes but both callers of blk_trace_setup_queue() actually check that q->blk_trace is NULL before calling blk_trace_setup_queue() and since we hold blk_trace_mutex all the time, the value of q->blk_trace cannot change. So the conditional assignment was just bogus.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 37 lines · Jan Kara
Quoted 12 lines · Bart Van Assche
(+Luis)
Quoted 9 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.Quoted 5 lines
Quoted 2 lines
Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>How about adding a reference to commit
c780e86dd48e("blktrace: Protectq->blk_tracewith RCU") in the description of this patch?Yes, that's probably a good idea.
Quoted 4 lines
Quoted 4 lines
kernel/trace/blktrace.c +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q,blk_trace_setup_lba (bt, bdev);Quoted 11 lines
kernel/trace/blktrace.c static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
kernel/trace/blktrace.c +1
-4- ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto free_bt; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref ();return 0 ;This changes a conditional assignment of
q->blk_traceinto an unconditional assignment. Shouldn'tq->blk_traceonly be assigned ifq->blk_trace== NULL?
Quoted 4 lines
Yes but both callers of
blk_trace_setup_queue()actually check thatq->blk_traceis NULL before callingblk_trace_setup_queue()and since we holdblk_trace_mutexall the time, the value ofq->blk_tracecannot change. So the conditional assignment was just bogus.
If you run a blktrace against a different partition the check does have an effect today. This is because the request_queue is shared between partitions implicitly, even though they end up using a different struct dentry. So the check is actually still needed, however my change adds this check early as well so we don't do a memory allocation just to throw it away.
Even then, the last final check is neede then.
Try these:
https://github.com/mcgrof/break-blktrace
Luis
Quoted 43 lines · Luis Chamberlain
Quoted 37 lines · Jan Kara
Quoted 12 lines · Bart Van Assche
(+Luis)
Quoted 9 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.Quoted 5 lines
Quoted 2 lines
Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>How about adding a reference to commit
c780e86dd48e("blktrace: Protectq->blk_tracewith RCU") in the description of this patch?Yes, that's probably a good idea.
Quoted 4 lines
Quoted 4 lines
kernel/trace/blktrace.c +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q,blk_trace_setup_lba (bt, bdev);Quoted 11 lines
kernel/trace/blktrace.c static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
kernel/trace/blktrace.c +1
-4- ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto free_bt; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref ();return 0 ;This changes a conditional assignment of
q->blk_traceinto an unconditional assignment. Shouldn'tq->blk_traceonly be assigned ifq->blk_trace== NULL?Quoted 4 lines
Yes but both callers of
blk_trace_setup_queue()actually check thatq->blk_traceis NULL before callingblk_trace_setup_queue()and since we holdblk_trace_mutexall the time, the value ofq->blk_tracecannot change. So the conditional assignment was just bogus.
Quoted 6 lines
If you run a blktrace against a different partition the check does have an effect today. This is because the
request_queueis shared between partitions implicitly, even though they end up using a different struct dentry. So the check is actually still needed, however my change adds this check early as well so we don't do a memory allocation just to throw it away.
I'm not sure we are speaking about the same check but I might be missing something. blk_trace_setup_queue() is only called from sysfs_blk_trace_attr_store(). That does:
mutex_lock(&q->blk_trace_mutex);
bt = rcu_dereference_protected(q->blk_trace,
lockdep_is_held(&q->blk_trace_mutex));
if (attr == &dev_attr_enable) {
if (!!value == !!bt) {
ret = 0;
goto out_unlock_bdev;
}
^^^ So if 'bt' is non-NULL, and we are enabling, we bailinstead of calling blk_trace_setup_queue().
Similarly later:
if (bt == NULL) {
ret = blk_trace_setup_queue(q, bdev);
...so we again call blk_trace_setup_queue() only if bt is NULL. So IMO the cmpxchg() in blk_trace_setup_queue() could never fail to set the value. Am I missing something?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 68 lines · Jan Kara
Quoted 43 lines · Luis Chamberlain
Quoted 37 lines · Jan Kara
Quoted 12 lines · Bart Van Assche
(+Luis)
Quoted 9 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.Quoted 5 lines
Quoted 2 lines
Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>How about adding a reference to commit
c780e86dd48e("blktrace: Protectq->blk_tracewith RCU") in the description of this patch?Yes, that's probably a good idea.
Quoted 4 lines
Quoted 4 lines
kernel/trace/blktrace.c +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q,blk_trace_setup_lba (bt, bdev);Quoted 11 lines
kernel/trace/blktrace.c static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
kernel/trace/blktrace.c +1
-4- ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto free_bt; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref ();return 0 ;This changes a conditional assignment of
q->blk_traceinto an unconditional assignment. Shouldn'tq->blk_traceonly be assigned ifq->blk_trace== NULL?Quoted 4 lines
Yes but both callers of
blk_trace_setup_queue()actually check thatq->blk_traceis NULL before callingblk_trace_setup_queue()and since we holdblk_trace_mutexall the time, the value ofq->blk_tracecannot change. So the conditional assignment was just bogus.Quoted 6 lines
If you run a blktrace against a different partition the check does have an effect today. This is because the
request_queueis shared between partitions implicitly, even though they end up using a different struct dentry. So the check is actually still needed, however my change adds this check early as well so we don't do a memory allocation just to throw it away.I'm not sure we are speaking about the same check but I might be missing something.
blk_trace_setup_queue()is only called fromsysfs_blk_trace_attr_store(). That does:mutex_lock(&q->blk_trace_mutex); bt = rcu_dereference_protected(q->blk_trace, lockdep_is_held(&q->blk_trace_mutex)); if (attr == &dev_attr_enable) { if (!!value == !!bt) { ret = 0; goto out_unlock_bdev; } ^^^ So if 'bt' is non-NULL, and we are enabling, we bailinstead of calling
blk_trace_setup_queue().
Quoted 8 lines
Similarly later:
if (bt == NULL) { ret = blk_trace_setup_queue(q, bdev); ...so we again call
blk_trace_setup_queue()only if bt is NULL. So IMO thecmpxchg()inblk_trace_setup_queue()could never fail to set the value. Am I missing something?
I believe we are talking about the same check indeed. Consider the situation not as a race, but instead consider the state machine of the ioctl. The BLKTRACESETUP goes first, and when that is over we have not ran BLKTRACESTART. So, prior to BLKTRACESTART we can have another BLKTRACESETUP run but against another partition. At that point we have two cases trying to use the same request_queue and the same q->blk_trace, even though this was well protected with the mutex.
And so the final check is needed to ensure we only give one of the users the right to blktrace.
Did I misunderstand the check though?
Luis
Quoted 78 lines · Luis Chamberlain
Quoted 68 lines · Jan Kara
Quoted 43 lines · Luis Chamberlain
Quoted 37 lines · Jan Kara
Quoted 12 lines · Bart Van Assche
(+Luis)
Quoted 9 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.Quoted 5 lines
Quoted 2 lines
Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>How about adding a reference to commit
c780e86dd48e("blktrace: Protectq->blk_tracewith RCU") in the description of this patch?Yes, that's probably a good idea.
Quoted 4 lines
Quoted 4 lines
kernel/trace/blktrace.c +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q,blk_trace_setup_lba (bt, bdev);Quoted 11 lines
kernel/trace/blktrace.c static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
kernel/trace/blktrace.c +1
-4- ret = -EBUSY ; -if (cmpxchg (&q->blk_trace,NULL , bt)) - goto free_bt; - +rcu_assign_pointer (q->blk_trace, bt);get_probe_ref ();return 0 ;This changes a conditional assignment of
q->blk_traceinto an unconditional assignment. Shouldn'tq->blk_traceonly be assigned ifq->blk_trace== NULL?Quoted 4 lines
Yes but both callers of
blk_trace_setup_queue()actually check thatq->blk_traceis NULL before callingblk_trace_setup_queue()and since we holdblk_trace_mutexall the time, the value ofq->blk_tracecannot change. So the conditional assignment was just bogus.Quoted 6 lines
If you run a blktrace against a different partition the check does have an effect today. This is because the
request_queueis shared between partitions implicitly, even though they end up using a different struct dentry. So the check is actually still needed, however my change adds this check early as well so we don't do a memory allocation just to throw it away.I'm not sure we are speaking about the same check but I might be missing something.
blk_trace_setup_queue()is only called fromsysfs_blk_trace_attr_store(). That does:mutex_lock(&q->blk_trace_mutex); bt = rcu_dereference_protected(q->blk_trace, lockdep_is_held(&q->blk_trace_mutex)); if (attr == &dev_attr_enable) { if (!!value == !!bt) { ret = 0; goto out_unlock_bdev; } ^^^ So if 'bt' is non-NULL, and we are enabling, we bailinstead of calling
blk_trace_setup_queue().Quoted 8 lines
Similarly later:
if (bt == NULL) { ret = blk_trace_setup_queue(q, bdev); ...so we again call
blk_trace_setup_queue()only if bt is NULL. So IMO thecmpxchg()inblk_trace_setup_queue()could never fail to set the value. Am I missing something?
Quoted 5 lines
I believe we are talking about the same check indeed. Consider the situation not as a race, but instead consider the state machine of the ioctl. The BLKTRACESETUP goes first, and when that is over we have not ran BLKTRACESTART. So, prior to BLKTRACESTART we can have another BLKTRACESETUP run but against another partition.
So first note that BLKTRACESETUP goes through do_blk_trace_setup() while 'echo 1 >/sys/block/../trace/enable' goes through blk_trace_setup_queue(). Although these operations achieve a very similar things, they are completely separate code paths. I was speaking about the second case while you are now speaking about the first one.
WRT to your BLKTRACESETUP example, the first BLKTRACESETUP will end up setting q->blk_trace to 'bt' so the second BLKTRACESETUP will see q->blk_trace is not NULL (my patch adds this check to do_blk_trace_setup() so we bail out earlier than during cmpxchg()) and fails. Again I don't see any problem here...
Quoted 7 lines
At that
point we have two cases trying to use the samerequest_queueand the sameq->blk_trace, even though this was well protected with the mutex.And so the final check is needed to ensure we only give one of the users the right to blktrace.
Well, the check to make sure there's only one trace setup is the one checking q->blk_trace is NULL while holding q->blk_trace_mutex...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 91 lines · Jan Kara
Quoted 78 lines · Luis Chamberlain
Quoted 68 lines · Jan Kara
Quoted 43 lines · Luis Chamberlain
Quoted 37 lines · Jan Kara
Quoted 12 lines · Bart Van Assche
(+Luis)
Quoted 9 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.Quoted 5 lines
Quoted 2 lines
Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>How about adding a reference to commit
c780e86dd48e("blktrace: Protectq->blk_tracewith RCU") in the description of this patch?Yes, that's probably a good idea.
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q, blk_trace_setup_lba(bt, bdev);Quoted 11 lines
Review static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
(quoted diff) +1
-4- ret = -EBUSY; - if (cmpxchg(&q->blk_trace, NULL, bt)) - goto free_bt; - + rcu_assign_pointer(q->blk_trace, bt); get_probe_ref(); return 0;This changes a conditional assignment of
q->blk_traceinto an unconditional assignment. Shouldn'tq->blk_traceonly be assigned ifq->blk_trace== NULL?Quoted 4 lines
Yes but both callers of
blk_trace_setup_queue()actually check thatq->blk_traceis NULL before callingblk_trace_setup_queue()and since we holdblk_trace_mutexall the time, the value ofq->blk_tracecannot change. So the conditional assignment was just bogus.Quoted 6 lines
If you run a blktrace against a different partition the check does have an effect today. This is because the
request_queueis shared between partitions implicitly, even though they end up using a different struct dentry. So the check is actually still needed, however my change adds this check early as well so we don't do a memory allocation just to throw it away.I'm not sure we are speaking about the same check but I might be missing something.
blk_trace_setup_queue()is only called fromsysfs_blk_trace_attr_store(). That does:mutex_lock(&q->blk_trace_mutex); bt = rcu_dereference_protected(q->blk_trace, lockdep_is_held(&q->blk_trace_mutex)); if (attr == &dev_attr_enable) { if (!!value == !!bt) { ret = 0; goto out_unlock_bdev; } ^^^ So if 'bt' is non-NULL, and we are enabling, we bailinstead of calling
blk_trace_setup_queue().Quoted 8 lines
Similarly later:
if (bt == NULL) { ret = blk_trace_setup_queue(q, bdev); ...so we again call
blk_trace_setup_queue()only if bt is NULL. So IMO thecmpxchg()inblk_trace_setup_queue()could never fail to set the value. Am I missing something?Quoted 5 lines
I believe we are talking about the same check indeed. Consider the situation not as a race, but instead consider the state machine of the ioctl. The BLKTRACESETUP goes first, and when that is over we have not ran BLKTRACESTART. So, prior to BLKTRACESTART we can have another BLKTRACESETUP run but against another partition.
So first note that BLKTRACESETUP goes through
do_blk_trace_setup()while 'echo 1 >/sys/block/../trace/enable' goes throughblk_trace_setup_queue(). Although these operations achieve a very similar things, they are completely separate code paths. I was speaking about the second case while you are now speaking about the first one.
Quoted 5 lines
WRT to your BLKTRACESETUP example, the first BLKTRACESETUP will end up setting
q->blk_traceto 'bt' so the second BLKTRACESETUP will seeq->blk_traceis not NULL (my patch adds this check todo_blk_trace_setup()so we bail out earlier than duringcmpxchg()) and fails. Again I don't see any problem here...
Ah, the patch I was CC'd on didn't contain this hunk! It only had the change from cmpxchg() to the rcu_assign_pointer(), so I misunderstood your intention, sorry!
In that case, I already proposed a patch to do that, and it also adds a tiny bit of verbiage given we currently don't inform the user about why this fails [0].
Let me know how you folks would like to proceed.
[0] https://lkml.kernel.org/r/[email protected]
Luis
Quoted 98 lines · Luis Chamberlain
Quoted 91 lines · Jan Kara
Quoted 78 lines · Luis Chamberlain
Quoted 68 lines · Jan Kara
Quoted 43 lines · Luis Chamberlain
Quoted 37 lines · Jan Kara
Quoted 12 lines · Bart Van Assche
(+Luis)
Quoted 9 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.Quoted 5 lines
Quoted 2 lines
Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>How about adding a reference to commit
c780e86dd48e("blktrace: Protectq->blk_tracewith RCU") in the description of this patch?Yes, that's probably a good idea.
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q, blk_trace_setup_lba(bt, bdev);Quoted 11 lines
Review static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
(quoted diff) +1
-4- ret = -EBUSY; - if (cmpxchg(&q->blk_trace, NULL, bt)) - goto free_bt; - + rcu_assign_pointer(q->blk_trace, bt); get_probe_ref(); return 0;This changes a conditional assignment of
q->blk_traceinto an unconditional assignment. Shouldn'tq->blk_traceonly be assigned ifq->blk_trace== NULL?Quoted 4 lines
Yes but both callers of
blk_trace_setup_queue()actually check thatq->blk_traceis NULL before callingblk_trace_setup_queue()and since we holdblk_trace_mutexall the time, the value ofq->blk_tracecannot change. So the conditional assignment was just bogus.Quoted 6 lines
If you run a blktrace against a different partition the check does have an effect today. This is because the
request_queueis shared between partitions implicitly, even though they end up using a different struct dentry. So the check is actually still needed, however my change adds this check early as well so we don't do a memory allocation just to throw it away.I'm not sure we are speaking about the same check but I might be missing something.
blk_trace_setup_queue()is only called fromsysfs_blk_trace_attr_store(). That does:mutex_lock(&q->blk_trace_mutex); bt = rcu_dereference_protected(q->blk_trace, lockdep_is_held(&q->blk_trace_mutex)); if (attr == &dev_attr_enable) { if (!!value == !!bt) { ret = 0; goto out_unlock_bdev; } ^^^ So if 'bt' is non-NULL, and we are enabling, we bailinstead of calling
blk_trace_setup_queue().Quoted 8 lines
Similarly later:
if (bt == NULL) { ret = blk_trace_setup_queue(q, bdev); ...so we again call
blk_trace_setup_queue()only if bt is NULL. So IMO thecmpxchg()inblk_trace_setup_queue()could never fail to set the value. Am I missing something?Quoted 5 lines
I believe we are talking about the same check indeed. Consider the situation not as a race, but instead consider the state machine of the ioctl. The BLKTRACESETUP goes first, and when that is over we have not ran BLKTRACESTART. So, prior to BLKTRACESTART we can have another BLKTRACESETUP run but against another partition.
So first note that BLKTRACESETUP goes through
do_blk_trace_setup()while 'echo 1 >/sys/block/../trace/enable' goes throughblk_trace_setup_queue(). Although these operations achieve a very similar things, they are completely separate code paths. I was speaking about the second case while you are now speaking about the first one.Quoted 5 lines
WRT to your BLKTRACESETUP example, the first BLKTRACESETUP will end up setting
q->blk_traceto 'bt' so the second BLKTRACESETUP will seeq->blk_traceis not NULL (my patch adds this check todo_blk_trace_setup()so we bail out earlier than duringcmpxchg()) and fails. Again I don't see any problem here...
Quoted 3 lines
Ah, the patch I was CC'd on didn't contain this hunk! It only had the change from
cmpxchg()to thercu_assign_pointer(), so I misunderstood your intention, sorry!
Good that we are on the same page now :)
Quoted 3 lines
In that case, I already proposed a patch to do that, and it also adds a tiny bit of verbiage given we currently don't inform the user about why this fails [0].
Honestly, I'm not sure pr_warn() you've added is that useful. We usually don't spam logs due to someone trying to use already used resource. But anyway, I can see other people are fine with that so I don't insist.
Quoted 1 line
Let me know how you folks would like to proceed.
I guess I can rebase my patch on top of your series since that seems pretty much done. I was aware of it but didn't realize there's a conflict...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 99 lines · Jan Kara
Quoted 98 lines · Luis Chamberlain
Quoted 96 lines · Jan Kara
Quoted 78 lines · Luis Chamberlain
Quoted 68 lines · Jan Kara
Quoted 43 lines · Luis Chamberlain
Quoted 37 lines · Jan Kara
Quoted 12 lines · Bart Van Assche
(+Luis)
Quoted 9 lines · Jan Kara
Mostly for historical reasons,
q->blk_traceis assigned throughxchg()andcmpxchg()atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations. Furthermore there's no real need for atomic operations anymore since all changes toq->blk_tracehappen underq->blk_trace_mutex. So let's just replacexchg()withrcu_replace_pointer()andcmpxchg()with explicit check andrcu_assign_pointer(). This makes the code more efficient and sparse happy.Quoted 5 lines
Quoted 2 lines
Reported-by kbuild test robot <[email protected]>Signed-off-by Jan Kara <[email protected]>How about adding a reference to commit
c780e86dd48e("blktrace: Protectq->blk_tracewith RCU") in the description of this patch?Yes, that's probably a good idea.
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-0@@ -1669,10 +1672,7 @@ static int blk_trace_setup_queue(struct request_queue *q, blk_trace_setup_lba(bt, bdev);Quoted 11 lines
Review static int blk_trace_setup_queue(struct request_queue *q, · line 1677
Quoted 7 lines
(quoted diff) +1
-4- ret = -EBUSY; - if (cmpxchg(&q->blk_trace, NULL, bt)) - goto free_bt; - + rcu_assign_pointer(q->blk_trace, bt); get_probe_ref(); return 0;This changes a conditional assignment of
q->blk_traceinto an unconditional assignment. Shouldn'tq->blk_traceonly be assigned ifq->blk_trace== NULL?Quoted 4 lines
Yes but both callers of
blk_trace_setup_queue()actually check thatq->blk_traceis NULL before callingblk_trace_setup_queue()and since we holdblk_trace_mutexall the time, the value ofq->blk_tracecannot change. So the conditional assignment was just bogus.Quoted 6 lines
If you run a blktrace against a different partition the check does have an effect today. This is because the
request_queueis shared between partitions implicitly, even though they end up using a different struct dentry. So the check is actually still needed, however my change adds this check early as well so we don't do a memory allocation just to throw it away.I'm not sure we are speaking about the same check but I might be missing something.
blk_trace_setup_queue()is only called fromsysfs_blk_trace_attr_store(). That does:mutex_lock(&q->blk_trace_mutex); bt = rcu_dereference_protected(q->blk_trace, lockdep_is_held(&q->blk_trace_mutex)); if (attr == &dev_attr_enable) { if (!!value == !!bt) { ret = 0; goto out_unlock_bdev; } ^^^ So if 'bt' is non-NULL, and we are enabling, we bailinstead of calling
blk_trace_setup_queue().Quoted 8 lines
Similarly later:
if (bt == NULL) { ret = blk_trace_setup_queue(q, bdev); ...so we again call
blk_trace_setup_queue()only if bt is NULL. So IMO thecmpxchg()inblk_trace_setup_queue()could never fail to set the value. Am I missing something?Quoted 5 lines
I believe we are talking about the same check indeed. Consider the situation not as a race, but instead consider the state machine of the ioctl. The BLKTRACESETUP goes first, and when that is over we have not ran BLKTRACESTART. So, prior to BLKTRACESTART we can have another BLKTRACESETUP run but against another partition.
So first note that BLKTRACESETUP goes through
do_blk_trace_setup()while 'echo 1 >/sys/block/../trace/enable' goes throughblk_trace_setup_queue(). Although these operations achieve a very similar things, they are completely separate code paths. I was speaking about the second case while you are now speaking about the first one.WRT to your BLKTRACESETUP example, the first BLKTRACESETUP will end up setting
q->blk_traceto 'bt' so the second BLKTRACESETUP will seeq->blk_traceis not NULL (my patch adds this check todo_blk_trace_setup()so we bail out earlier than duringcmpxchg()) and fails. Again I don't see any problem here...
Quoted 5 lines
Quoted 3 lines
Ah, the patch I was CC'd on didn't contain this hunk! It only had the change from
cmpxchg()to thercu_assign_pointer(), so I misunderstood your intention, sorry!Good that we are on the same page now :)
Yay!
Quoted 7 lines
Quoted 3 lines
In that case, I already proposed a patch to do that, and it also adds a tiny bit of verbiage given we currently don't inform the user about why this fails [0].
Honestly, I'm not sure
pr_warn()you've added is that useful. We usually don't spam logs due to someone trying to use already used resource. But anyway, I can see other people are fine with that so I don't insist.
Well I would typically agree... however...
It is in no way shape or form, not even in the blktrace documentation that the request_queue / and therefore blktrace is shared between partitions. Likewise for scsi-generic and say its respective block device for TYPE_BLOCK.
If it is not obvious to some developer, it won't be obvious to users. So *why* this fails really today is a mystery to users.
These limitations to the design of blktrace is not well documented at all.
Quoted 4 lines
Quoted 1 line
Let me know how you folks would like to proceed.
I guess I can rebase my patch on top of your series since that seems pretty much done.
I think so as well.
Quoted 1 line
I was aware of it but didn't realize there's a conflict...
Thanks for Bart for pointing it out!
Luis