Skip to contentSkip to conversation

blktrace: Fix sparse annotations and warn if enabling multiple traces

PATCH In review Jan Kara 4 participants 10 messages Jun 5, 2020 – Jun 17, 2020

Subsystems (2)
lore
Share & tools
lore

Jan Kara

author
#0 lore raw

v3 cover blktrace: Fix sparse annotations and warn if enabling multiple traces

Hello Jens,

this series contains a patch from Luis' blktrace series and then my patch to fix sparse warnings in blktrace. Luis' patch stands on its own, was reviewed, and changes what I need to change as well so I've decided it's just simplest to pull it in with my patch.

Honza

author
#1 lore raw

PATCH v3 1/2 blktrace: break out of blktrace setup on concurrent calls

From: Luis Chamberlain <[email protected]>

We use one blktrace per request_queue, that means one per the entire disk. So we cannot run one blktrace on say /dev/vda and then /dev/vda1, or just two calls on /dev/vda.

We check for concurrent setup only at the very end of the blktrace setup though.

If we try to run two concurrent blktraces on the same block device the second one will fail, and the first one seems to go on. However when one tries to kill the first one one will see things like this:

The kernel will show these:

```
debugfs: File 'dropped' in directory 'nvme1n1' already present!
debugfs: File 'msg' in directory 'nvme1n1' already present!
debugfs: File 'trace0' in directory 'nvme1n1' already present!
``

And userspace just sees this error message for the second call:

```

blktrace /dev/nvme1n1
BLKTRACESETUP(2) /dev/nvme1n1 failed: 5/Input/output error ```

The first userspace process #1 will also claim that the files were taken underneath their nose as well. The files are taken away form the first process given that when the second blktrace fails, it will follow up with a BLKTRACESTOP and BLKTRACETEARDOWN. This means that even if go-happy process #1 is waiting for blktrace data, we *have* been asked to take teardown the blktrace.

This can easily be reproduced with break-blktrace [0] run_0005.sh test.

Just break out early if we know we're already going to fail, this will prevent trying to create the files all over again, which we know still exist.

[0] https://github.com/mcgrof/break-blktrace

---
 kernel/trace/blktrace.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/trace/blktrace.c +13 -0
index ea47f2084087..1a1943d39802 100644
@@ -3,6 +3,9 @@
  * Copyright (C) 2006 Jens Axboe <[email protected]>
  *
  */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
 #include <linux/kernel.h>
 #include <linux/blkdev.h>
 #include <linux/blktrace_api.h>
@@ -494,6 +497,16 @@ static int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 	 */
 	strreplace(buts->name, '/', '_');
 
+	/*
+	 * bdev can be NULL, as with scsi-generic, this is a helpful as
+	 * we can be.
+	 */
+	if (q->blk_trace) {
+		pr_warn("Concurrent blktraces are not allowed on %s\n",
+			buts->name);
+		return -EBUSY;
+	}
+
 	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
 	if (!bt)
 		return -ENOMEM;
Signature · 1 line
2.16.4
Reviewed-by on v3 1/2
#2 lore raw

Re: [PATCH 1/2] blktrace: break out of blktrace setup on concurrent calls

Quoted 14 lines · Jan Kara
kernel/trace/blktrace.c +7 -0
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
  #include <linux/kernel.h>
  #include <linux/blkdev.h>
  #include <linux/blktrace_api.h>
@@ -494,6 +497,16 @@ static int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
  	 */
  	strreplace(buts->name, '/', '_');
 
+	/*
+	 * bdev can be NULL, as with scsi-generic, this is a helpful as
+	 * we can be.
+	 */

kernel/trace/blktrace.c static int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,

Quoted 10 lines
kernel/trace/blktrace.c +6 -1
+	if (q->blk_trace) {
+		pr_warn("Concurrent blktraces are not allowed on %s\n",
+			buts->name);
+		return -EBUSY;
+	}
+
  	bt = kzalloc(sizeof(*bt), GFP_KERNEL);
  	if (!bt)
  		return -ENOMEM;
-- 2.16.4

Looks good.

author
#3 lore raw

PATCH v3 2/2 blktrace: Avoid sparse warnings when assigning q->blk_trace

In reply to Jan Kara

Mostly for historical reasons, q->blk_trace is assigned through xchg() and cmpxchg() atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations since commit c780e86dd48e ("blktrace: Protect q->blk_trace with RCU") which started to use rcu for accessing q->blk_trace. Furthermore there's no real need for atomic operations anymore since all changes to q->blk_trace happen under q->blk_trace_mutex and since it also makes more sense to check if q->blk_trace is set with the mutex held earlier.

So let's just replace xchg() with rcu_replace_pointer() and cmpxchg() with explicit check and rcu_assign_pointer(). This makes the code more efficient and sparse happy.

---
 kernel/trace/blktrace.c | 19 ++++++++-----------
 1 file changed, 8 insertions(+), 11 deletions(-)
kernel/trace/blktrace.c +8 -11
index 1a1943d39802..1b8e2eaf23f7 100644
@@ -347,7 +347,8 @@ static int __blk_trace_remove(struct request_queue *q)
 {
 	struct blk_trace *bt;
 
-	bt = xchg(&q->blk_trace, NULL);
+	bt = rcu_replace_pointer(q->blk_trace, NULL,
+				 lockdep_is_held(&q->blk_trace_mutex));
 	if (!bt)
 		return -EINVAL;
 
@@ -501,7 +502,8 @@ static int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 	 * bdev can be NULL, as with scsi-generic, this is a helpful as
 	 * we can be.
 	 */
-	if (q->blk_trace) {
+	if (rcu_dereference_protected(q->blk_trace,
+				      lockdep_is_held(&q->blk_trace_mutex))) {
 		pr_warn("Concurrent blktraces are not allowed on %s\n",
 			buts->name);
 		return -EBUSY;
@@ -556,10 +558,7 @@ static int do_blk_trace_setup(struct request_queue *q, char *name, dev_t dev,
 	bt->pid = buts->pid;
 	bt->trace_state = Blktrace_setup;
 
-	ret = -EBUSY;
-	if (cmpxchg(&q->blk_trace, NULL, bt))
-		goto err;
-
+	rcu_assign_pointer(q->blk_trace, bt);
 	get_probe_ref();
 
 	ret = 0;
@@ -1650,7 +1649,8 @@ static int blk_trace_remove_queue(struct request_queue *q)
 {
 	struct blk_trace *bt;
 
-	bt = xchg(&q->blk_trace, NULL);
+	bt = rcu_replace_pointer(q->blk_trace, NULL,
+				 lockdep_is_held(&q->blk_trace_mutex));
 	if (bt == NULL)
 		return -EINVAL;
 
@@ -1682,10 +1682,7 @@ static int blk_trace_setup_queue(struct request_queue *q,
 
 	blk_trace_setup_lba(bt, bdev);
 
-	ret = -EBUSY;
-	if (cmpxchg(&q->blk_trace, NULL, bt))
-		goto free_bt;
-
+	rcu_assign_pointer(q->blk_trace, bt);
 	get_probe_ref();
 	return 0;
 
Signature · 1 line
2.16.4
Reviewed-by on v3 2/2
#4 lore raw

Re: [PATCH 2/2] blktrace: Avoid sparse warnings when assigning q->blk_trace

Quoted 9 lines · Jan Kara

Mostly for historical reasons, q->blk_trace is assigned through xchg() and cmpxchg() atomic operations. Although this is correct, sparse complains about this because it violates rcu annotations since commit c780e86dd48e ("blktrace: Protect q->blk_trace with RCU") which started to use rcu for accessing q->blk_trace. Furthermore there's no real need for atomic operations anymore since all changes to q->blk_trace happen under q->blk_trace_mutex and since it also makes more sense to check if q->blk_trace is set with the mutex held earlier.

Quoted 6 lines

So let's just replace xchg() with rcu_replace_pointer() and cmpxchg() with explicit check and rcu_assign_pointer(). This makes the code more efficient and sparse happy.

Looks good.

#5 lore raw
In reply to Jan Kara
Quoted 6 lines · Jan Kara

Hello Jens,

this series contains a patch from Luis' blktrace series and then my patch to fix sparse warnings in blktrace. Luis' patch stands on its own, was reviewed, and changes what I need to change as well so I've decided it's just simplest to pull it in with my patch.

Hey Jan,

Since these patches have contexts which can affect other patches in the series of fixes I have, if possible I'd prefer if these get addressed in order. I was just waiting for 0-day to finish grinding on the series. I already have a successful build results, but just waiting on the series of other tests, part of which include blktests.

If this is agreeable, I hope to post that series later today.

Luis

author
#6 lore raw

Hi Luis!

Quoted 10 lines · Luis Chamberlain
Quoted 6 lines · Jan Kara

Hello Jens,

this series contains a patch from Luis' blktrace series and then my patch to fix sparse warnings in blktrace. Luis' patch stands on its own, was reviewed, and changes what I need to change as well so I've decided it's just simplest to pull it in with my patch.

Hey Jan,

Quoted 7 lines

Since these patches have contexts which can affect other patches in the series of fixes I have, if possible I'd prefer if these get addressed in order. I was just waiting for 0-day to finish grinding on the series. I already have a successful build results, but just waiting on the series of other tests, part of which include blktests.

If this is agreeable, I hope to post that series later today.

Whatever... I wasn't sure how long it's going to take to finish your series and since Jens was asking about the sparse fixes, I've just sent them along. BTW the two patches applied just fine separately from the rest of the series so I don't think there's any dependency to the rest of your series even in context.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#7 lore raw
Quoted 3 lines · Jan Kara

BTW the two patches applied just fine separately from the rest of the series so I don't think there's any dependency to the rest of your series even in context.

The depndency was just minor contextual edits, which can be fixed with a rebase, I've done that now, so this is good to go in first. I'll skip these two patches fromy series then.

Luis

author
#8 lore raw
In reply to Jan Kara

Hello Jens!

Quoted 4 lines · Jan Kara

this series contains a patch from Luis' blktrace series and then my patch to fix sparse warnings in blktrace. Luis' patch stands on its own, was reviewed, and changes what I need to change as well so I've decided it's just simplest to pull it in with my patch.

Can you please merge this series?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#9 lore raw
Quoted 9 lines · Jan Kara

Hello Jens!

Quoted 4 lines · Jan Kara

this series contains a patch from Luis' blktrace series and then my patch to fix sparse warnings in blktrace. Luis' patch stands on its own, was reviewed, and changes what I need to change as well so I've decided it's just simplest to pull it in with my patch.

Can you please merge this series?

Yep done, thanks.

Signature · 1 line
Jens Axboe