blk-mq-sched: improve SCSI-MQ performance

When hw queue is busy, we shouldn't take requests from scheduler queue any more, otherwise IO merge will be difficult to do.

This patch fixes the awful IO performance on some SCSI devices(lpfc, qla2xxx, ...) when mq-deadline/kyber is used by not taking requests if hw queue is busy.

Reviewed-by Bart Van Assche <[email protected]>

Signed-off-by Ming Lei <[email protected]>

---
 block/blk-mq-sched.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

block/blk-mq-sched.c +3 -3

index 4ab69435708c..845e5baf8af1 100644
@@ -94,7 +94,7 @@ void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
 	struct request_queue *q = hctx->queue;
 	struct elevator_queue *e = q->elevator;
 	const bool has_sched_dispatch = e && e->type->ops.mq.dispatch_request;
-	bool did_work = false;
+	bool do_sched_dispatch = true;
 	LIST_HEAD(rq_list);
 
 	/* RCU or SRCU read lock is needed before checking quiesced flag */

@@ -125,7 +125,7 @@ void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
 	 */
 	if (!list_empty(&rq_list)) {
 		blk_mq_sched_mark_restart_hctx(hctx);
-		did_work = blk_mq_dispatch_rq_list(q, &rq_list);
+		do_sched_dispatch = blk_mq_dispatch_rq_list(q, &rq_list);
 	} else if (!has_sched_dispatch) {
 		blk_mq_flush_busy_ctxs(hctx, &rq_list);
 		blk_mq_dispatch_rq_list(q, &rq_list);

@@ -136,7 +136,7 @@ void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
 	 * on the dispatch list, OR if we did have work but weren't able
 	 * to make progress.
 	 */
-	if (!did_work && has_sched_dispatch) {
+	if (do_sched_dispatch && has_sched_dispatch) {
 		do {
 			struct request *rq;
 

Signature · 1 line

2.9.4

Quoted 4 lines · Ming Lei

When hw queue is busy, we shouldn't take requests from scheduler queue any more, otherwise IO merge will be difficult to do.

Quoted 3 lines

This patch fixes the awful IO performance on some SCSI devices(lpfc, qla2xxx, ...) when mq-deadline/kyber is used by not taking requests if hw queue is busy.

Jens added this behavior in 64765a75ef25 ("blk-mq-sched: ask scheduler for work, if we failed dispatching leftovers"). That change was a big performance improvement, but we didn't figure out why. We'll need to dig up whatever test Jens was doing to make sure it doesn't regress.

Quoted 9 lines · Omar Sandoval

Quoted 7 lines · Ming Lei

When hw queue is busy, we shouldn't take requests from scheduler queue any more, otherwise IO merge will be difficult to do.

This patch fixes the awful IO performance on some SCSI devices(lpfc, qla2xxx, ...) when mq-deadline/kyber is used by not taking requests if hw queue is busy.

Quoted 4 lines

Jens added this behavior in 64765a75ef25 ("blk-mq-sched: ask scheduler for work, if we failed dispatching leftovers"). That change was a big performance improvement, but we didn't figure out why. We'll need to dig up whatever test Jens was doing to make sure it doesn't regress.

Not found info about Jen's test case on this commit from google.

Maybe Jens could provide some input about your test case?

In theory, if hw queue is busy and requests are left in ->dispatch, we should not have continued to dequeue requests from sw/scheduler queue any more. Otherwise, IO merge can be hurt much. At least on SCSI devices, this improved much on sequential I/O, at least 3X of sequential read is increased on lpfc with this patch, in case of mq-deadline.

Or are there other special cases in which we still need to push requests hard into a busy hardware?

And this patch won't have an effect on devices in which queue busy is seldom triggered, such as NVMe.

Quoted 10 lines · Ming Lei

Quoted 9 lines · Omar Sandoval

Quoted 7 lines · Ming Lei

When hw queue is busy, we shouldn't take requests from scheduler queue any more, otherwise IO merge will be difficult to do.

This patch fixes the awful IO performance on some SCSI devices(lpfc, qla2xxx, ...) when mq-deadline/kyber is used by not taking requests if hw queue is busy.

Quoted 8 lines

Quoted 4 lines

Jens added this behavior in 64765a75ef25 ("blk-mq-sched: ask scheduler for work, if we failed dispatching leftovers"). That change was a big performance improvement, but we didn't figure out why. We'll need to dig up whatever test Jens was doing to make sure it doesn't regress.

Not found info about Jen's test case on this commit from google.

Maybe Jens could provide some input about your test case?

Okay I found my previous discussion with Jens (it was an off-list discussion). The test case was xfs/297 from xfstests: after 64765a75ef25, the test went from taking ~300 seconds to ~200 seconds on his SCSI device.

Quoted 5 lines

In theory, if hw queue is busy and requests are left in ->dispatch, we should not have continued to dequeue requests from sw/scheduler queue any more. Otherwise, IO merge can be hurt much. At least on SCSI devices, this improved much on sequential I/O, at least 3X of sequential read is increased on lpfc with this patch, in case of mq-deadline.

Right, your patch definitely makes more sense intuitively.

Quoted 2 lines

Or are there other special cases in which we still need to push requests hard into a busy hardware?

xfs/297 does a lot of fsyncs and hence a lot of flushes, that could be the special case.

Quoted 15 lines · Ming Lei

Quoted 13 lines · Omar Sandoval

Quoted 7 lines · Ming Lei

When hw queue is busy, we shouldn't take requests from scheduler queue any more, otherwise IO merge will be difficult to do.

This patch fixes the awful IO performance on some SCSI devices(lpfc, qla2xxx, ...) when mq-deadline/kyber is used by not taking requests if hw queue is busy.

Jens added this behavior in 64765a75ef25 ("blk-mq-sched: ask scheduler for work, if we failed dispatching leftovers"). That change was a big performance improvement, but we didn't figure out why. We'll need to dig up whatever test Jens was doing to make sure it doesn't regress.

Quoted 3 lines

Not found info about Jen's test case on this commit from google.

Maybe Jens could provide some input about your test case?

Okay I found my previous discussion with Jens (it was an off-list discussion). The test case was xfs/297 from xfstests: after 64765a75ef25, the test went from taking ~300 seconds to ~200 seconds on his SCSI device.

Quoted 6 lines

In theory, if hw queue is busy and requests are left in ->dispatch, we should not have continued to dequeue requests from sw/scheduler queue any more. Otherwise, IO merge can be hurt much. At least on SCSI devices, this improved much on sequential I/O, at least 3X of sequential read is increased on lpfc with this patch, in case of mq-deadline.

Right, your patch definitely makes more sense intuitively.

Quoted 2 lines

Or are there other special cases in which we still need to push requests hard into a busy hardware?

xfs/297 does a lot of fsyncs and hence a lot of flushes, that could be the special case.

Quoted 6 lines · Ming Lei

/**
* sbitmap_for_each_set() - Iterate over each set bit in a &struct sbitmap.
+ * @off: Offset to iterate from
* @sb: Bitmap to iterate over.
* @fn: Callback. Should return true to continue or false to break early.
* @data: Pointer to pass to callback.

Using 'off' as the name for the new argument seems confusing to me since that argument starts from zero and is not an offset relative to anything. Please consider to use 'start' as the name for this argument.

include/linux/sbitmap.h typedef bool (*sb_for_each_fn)(struct sbitmap *, unsigned int, void *); · line 233

Quoted 9 lines

include/linux/sbitmap.h +4 -3

-static inline void sbitmap_for_each_set(struct sbitmap *sb, sb_for_each_fn fn,
-					void *data)
+static inline void __sbitmap_for_each_set(struct sbitmap *sb,
+					  unsigned int off,
+					  sb_for_each_fn fn, void *data)
 {
-	unsigned int i;
+	unsigned int index = SB_NR_TO_INDEX(sb, off);

Is it really useful to rename 'i' into 'index'? I think that change makes this patch harder to read than necessary.

Quoted 4 lines

include/linux/sbitmap.h +2 -0

+	unsigned int nr = SB_NR_TO_BIT(sb, off);
+	unsigned int scanned = 0;

include/linux/sbitmap.h typedef bool (*sb_for_each_fn)(struct sbitmap *, unsigned int, void *); · line 237

Quoted 4 lines

include/linux/sbitmap.h +1 -3

-	for (i = 0; i < sb->map_nr; i++) {
-		struct sbitmap_word *word = &sb->map[i];
-		unsigned int off, nr;
+	while (1) {

Sorry but this change looks incorrect to me. I think the following two tests have to be performed before the while loop starts to avoid triggering an out-of-bounds reference of sb->map[]:
* Whether or not sb->map_nr is zero.

* Whether or not index >= sb->map_nr. I propose to start iterating from the start of @sb in this case.

Additionally, the new loop in __sbitmap_for_each_set() looks more complicated and more fragile to me than necessary. How about using the code below? That

code needs one local variable less than your implementation.

static inline void __sbitmap_for_each_set(struct sbitmap *sb,
const unsigned int start, sb_for_each_fn fn, void *data)

{
	unsigned int i = start >> sb->shift;
	unsigned int nr = start & ((1 << sb->shift) - 1);
	bool cycled = false;

	if (!sb->map_nr)
		return;

	if (unlikely(i >= sb->map_nr)) {
		i = 0;
		nr = 0;

	}

	while (true) {
		struct sbitmap_word *word = &sb->map[i];
		unsigned int off;

		off = i << sb->shift;
		while (1) {
			nr = find_next_bit(&word->word, word->depth, nr);

			if (cycled && off + nr >= start)
				return;

			if (nr >= word->depth)
				break;

			if (!fn(sb, off + nr, data))
				return;

			nr++;
		}
		if (++i >= sb->map_nr) {
			cycled = true;
			i = 0;
		}
		nr = 0;
	}
}

Thanks,

Bart.

Quoted 5 lines · Ming Lei

Quoted 2 lines · Bart Van Assche

* Whether or not index >= sb->map_nr. I propose to start iterating from the start of @sb in this case.

It has been checked at the end of the loop.

That's not sufficient to avoid an out-of-bounds access if the start index is large. If __sbitmap_for_each_set() would accept values for the start index argument that result in index >= sb->map_nr then that will simplify code that accesses an sbitmap in a round-robin fashion.

Quoted 7 lines

Quoted 6 lines

	}

	while (true) {
		struct sbitmap_word *word = &sb->map[i];
		unsigned int off;

Quoted 1 line

Looks you removed the check on 'word->word'.

Yes, and I did that on purpose. If the start index refers to a word that is zero then the "if (word->word) continue;" code will cause the end-of-loop check to be skipped and hence will cause an infinite loop.

include/linux/sbitmap.h typedef bool (*sb_for_each_fn)(struct sbitmap *, unsigned int, void *); · line 232

Quoted 6 lines · Ming Lei

include/linux/sbitmap.h +3 -2

-static inline void sbitmap_for_each_set(struct sbitmap *sb, sb_for_each_fn fn,
-					void *data)
+static inline void __sbitmap_for_each_set(struct sbitmap *sb,
+					  unsigned int off,
+					  sb_for_each_fn fn, void *data)
 {

An additional comment: if a function name starts with a double underscore usually that either means that it should be called with a specific lock held or that it is an implementation function that should not be called by other modules. Since neither is the case for __sbitmap_for_each_set(), please consider to use another name for this function.

Quoted 3 lines · Ming Lei

More importantly, for some SCSI devices, driver
tags are host wide, and the number is quite big,
but each lun has very limited queue depth.

This may be the case but is not always the case. Another important use-case is one LUN per host and where the queue depth per LUN is identical to the number of host tags.

block/blk-mq.c void blk_mq_flush_busy_ctxs(struct blk_mq_hw_ctx *hctx, struct list_head *list) · line 871

Quoted 5 lines

block/blk-mq.c +4 -0

+struct request *blk_mq_dispatch_rq_from_ctx(struct blk_mq_hw_ctx *hctx,
+					    struct blk_mq_ctx *start)
+{
+	unsigned off = start ? start->index_hw : 0;

Please consider to rename this function into blk_mq_dispatch_rq_from_next_ctx() and to start from start->index_hw + 1 instead of start->index_hw. I think that will not only result in simpler but also in faster code.

Quoted 14 lines · Ming Lei

Quoted 8 lines · Bart Van Assche

Quoted 3 lines · Ming Lei

More importantly, for some SCSI devices, driver
tags are host wide, and the number is quite big,
but each lun has very limited queue depth.

This may be the case but is not always the case. Another important use-case is one LUN per host and where the queue depth per LUN is identical to the number of host tags.

This patchset won't hurt this case because the BUSY info is returned from driver. In this case, BLK_STS_RESOURCE should seldom be returned from .queue_rq generally.

Quoted 5 lines

Also one important fact is that once q->queue_depth is set, that means there is per-request_queue limit on pending I/Os, and the single LUN is just the special case which is covered by this whole patchset. We don't need to pay special attention in this special case at all.

The purpose of my comment was not to ask for further clarification but to report that the description of this patch is not correct.

Quoted 11 lines

Quoted 10 lines

block/blk-mq.c void blk_mq_flush_busy_ctxs(struct blk_mq_hw_ctx *hctx, struct list_head *list) · line 871

Quoted 6 lines

block/blk-mq.c +4 -0

+struct request *blk_mq_dispatch_rq_from_ctx(struct blk_mq_hw_ctx *hctx,
+					    struct blk_mq_ctx *start)
+{
+	unsigned off = start ? start->index_hw : 0;

Please consider to rename this function into blk_mq_dispatch_rq_from_next_ctx() and to start from start->index_hw + 1 instead of start->index_hw. I think that will not only result in simpler but also in faster code.

Quoted 6 lines

I believe this helper with blk_mq_next_ctx(hctx, rq->mq_ctx) together will be much simpler and easier to implement, and code can be much readable too.

blk_mq_dispatch_rq_from_next_ctx() is ugly and mixing two things together.

Sorry but I disagree with both of the above statements.

Quoted 1 line · Ming Lei

easy to cause queue busy becasue of the small

                           ^^^^^^^
                           because?

block/blk-mq-sched.c static bool blk_mq_sched_restart_hctx(struct blk_mq_hw_ctx *hctx) · line 96

Quoted 8 lines

block/blk-mq-sched.c +3 -3

-static void blk_mq_do_dispatch(struct request_queue *q,
-			       struct elevator_queue *e,
-			       struct blk_mq_hw_ctx *hctx)
+static inline void blk_mq_do_dispatch_sched(struct request_queue *q,
+					    struct elevator_queue *e,
+					    struct blk_mq_hw_ctx *hctx)
 {
 	LIST_HEAD(rq_list);

Why to declare this function "inline"? Are you sure that the compiler is not smart enough to decide on its own whether or not to inline this function?

Quoted 1 line · Ming Lei

easy to cause queue busy becasue of the small

                           ^^^^^^^
                           because?

block/blk-mq-sched.c static bool blk_mq_sched_restart_hctx(struct blk_mq_hw_ctx *hctx) · line 96

Quoted 8 lines

block/blk-mq-sched.c +3 -3

-static void blk_mq_do_dispatch(struct request_queue *q,
-			       struct elevator_queue *e,
-			       struct blk_mq_hw_ctx *hctx)
+static inline void blk_mq_do_dispatch_sched(struct request_queue *q,
+					    struct elevator_queue *e,
+					    struct blk_mq_hw_ctx *hctx)
 {
 	LIST_HEAD(rq_list);

Why to declare this function "inline"? Are you sure that the compiler is not smart enough to decide on its own whether or not to inline this function?

Quoted 25 lines

block/blk-mq-sched.c +24 -0

+static inline struct blk_mq_ctx *blk_mq_next_ctx(struct blk_mq_hw_ctx *hctx,
+						 struct blk_mq_ctx *ctx)
+{
+	unsigned idx = ctx->index_hw;
+
+	if (++idx == hctx->nr_ctx)
+		idx = 0;
+
+	return hctx->ctxs[idx];
+}
+
+static inline void blk_mq_do_dispatch_ctx(struct request_queue *q,
+					  struct blk_mq_hw_ctx *hctx)
+{
+	LIST_HEAD(rq_list);
+	struct blk_mq_ctx *ctx = NULL;
+
+	do {
+		struct request *rq;
+
+		rq = blk_mq_dispatch_rq_from_ctx(hctx, ctx);
+		if (!rq)
+			break;
+		list_add(&rq->queuelist, &rq_list);

block/blk-mq-sched.c static void blk_mq_do_dispatch(struct request_queue *q, · line 136

Quoted 5 lines

block/blk-mq-sched.c +5 -0

+
+		/* round robin for fair dispatch */
+		ctx = blk_mq_next_ctx(hctx, rq->mq_ctx);
+	} while (blk_mq_dispatch_rq_list(q, &rq_list));
+}

Please consider to move the blk_mq_next_ctx() functionality into blk_mq_dispatch_rq_from_ctx() as requested in a comment on a previous patch.

block/blk-mq-sched.c void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx) · line 175

Quoted 10 lines

void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx) {

block/blk-mq-sched.c +1 -1

 	struct request_queue *q = hctx->queue;

@@ -142,18 +172,31 @@ void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
 	if (!list_empty(&rq_list)) {
 		blk_mq_sched_mark_restart_hctx(hctx);
 		do_sched_dispatch = blk_mq_dispatch_rq_list(q, &rq_list);
-	} else if (!has_sched_dispatch) {
+	} else if (!has_sched_dispatch & !q->queue_depth) {

Please use "&&" instead of "&" to represent logical and.

Otherwise this patch looks fine to me.

Quoted 15 lines · Ming Lei

block/blk-mq-sched.c +15 -0

+static inline void blk_mq_do_dispatch_ctx(struct request_queue *q,
+					  struct blk_mq_hw_ctx *hctx)
+{
+	LIST_HEAD(rq_list);
+	struct blk_mq_ctx *ctx = NULL;
+
+	do {
+		struct request *rq;
+
+		rq = blk_mq_dispatch_rq_from_ctx(hctx, ctx);
+		if (!rq)
+			break;
+		list_add(&rq->queuelist, &rq_list);
+
+		/* round robin for fair dispatch */

block/blk-mq-sched.c static void blk_mq_do_dispatch(struct request_queue *q, · line 136

Quoted 3 lines

block/blk-mq-sched.c +3 -0

+		ctx = blk_mq_next_ctx(hctx, rq->mq_ctx);
+	} while (blk_mq_dispatch_rq_list(q, &rq_list));
+}

An additional question about this patch: shouldn't request dequeuing start at the software queue next to the last one from which a request got dequeued instead of always starting at the first software queue (struct blk_mq_ctx *ctx = NULL) to be truly round robin?

block/blk-mq-sched.c void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx) · line 193

Quoted 7 lines · Ming Lei

block/blk-mq-sched.c +7 -0

+	/*
+	 * Wherever DISPATCH_BUSY is set, blk_mq_run_hw_queue()
+	 * will be run to try to make progress, so it is always
+	 * safe to check the state here.
+	 */
+	if (test_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state))
+		return;

The comment above test_bit() is useful but does not explain the purpose of the early return. Is the purpose of the early return perhaps to serialize blk_mq_sched_dispatch_requests() calls? If so, please mention this.

During dispatching, we moved all requests from hctx->dispatch to one temporary list, then dispatch them one by one from this list. Unfortunately duirng this period, run queue from other contexts may think the queue is idle, then start to dequeue from sw/scheduler queue and still try to dispatch because ->dispatch is empty. This way hurts sequential I/O performance because requests are dequeued when lld queue is busy.

This patch introduces the state of BLK_MQ_S_DISPATCH_BUSY to make sure that request isn't dequeued until ->dispatch is flushed.

Quoted 8 lines · Ming Lei

During dispatching, we moved all requests from hctx->dispatch to one temporary list, then dispatch them one by one from this list. Unfortunately duirng this period, run queue from other contexts may think the queue is idle, then start to dequeue from sw/scheduler queue and still try to dispatch because ->dispatch is empty. This way hurts sequential I/O performance because requests are dequeued when lld queue is busy.

Quoted 3 lines

This patch introduces the state of BLK_MQ_S_DISPATCH_BUSY to make sure that request isn't dequeued until ->dispatch is flushed.

I don't like how this patch introduces a bunch of locked setting of a flag under the hctx lock. Especially since I think we can easily avoid it.

Quoted 4 lines

block/blk-mq-sched.c +2 -1

-	} else if (!has_sched_dispatch & !q->queue_depth) {
+		blk_mq_dispatch_rq_list(q, &rq_list);
+

block/blk-mq-sched.c void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx) · line 184

Quoted 10 lines

block/blk-mq-sched.c +10 -0

+		/*
+		 * We may clear DISPATCH_BUSY just after it
+		 * is set from another context, the only cost
+		 * is that one request is dequeued a bit early,
+		 * we can survive that. Given the window is
+		 * too small, no need to worry about performance
+		 * effect.
+		 */
+		if (list_empty_careful(&hctx->dispatch))
+			clear_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);

This is basically the only place where we modify it without holding the hctx lock. Can we move it into blk_mq_dispatch_rq_list()? The list is

generally empty, unless for the case where we splice residuals back. If we splice them back, we grab the lock anyway.

The other places it's set under the hctx lock, yet we end up using an atomic operation to do it.

Quoted 13 lines · Jens Axboe

Quoted 11 lines · Ming Lei

During dispatching, we moved all requests from hctx->dispatch to one temporary list, then dispatch them one by one from this list. Unfortunately duirng this period, run queue from other contexts may think the queue is idle, then start to dequeue from sw/scheduler queue and still try to dispatch because ->dispatch is empty. This way hurts sequential I/O performance because requests are dequeued when lld queue is busy.

This patch introduces the state of BLK_MQ_S_DISPATCH_BUSY to make sure that request isn't dequeued until ->dispatch is flushed.

Quoted 3 lines

I don't like how this patch introduces a bunch of locked setting of a flag under the hctx lock. Especially since I think we can easily avoid it.

Actually the lock isn't needed for setting the flag, will move it out in V3.

Quoted 18 lines

Quoted 5 lines

block/blk-mq-sched.c +2 -1

-	} else if (!has_sched_dispatch & !q->queue_depth) {
+		blk_mq_dispatch_rq_list(q, &rq_list);
+

block/blk-mq-sched.c void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx) · line 184

Quoted 10 lines

block/blk-mq-sched.c +10 -0

+		/*
+		 * We may clear DISPATCH_BUSY just after it
+		 * is set from another context, the only cost
+		 * is that one request is dequeued a bit early,
+		 * we can survive that. Given the window is
+		 * too small, no need to worry about performance
+		 * effect.
+		 */
+		if (list_empty_careful(&hctx->dispatch))
+			clear_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);

This is basically the only place where we modify it without holding the hctx lock. Can we move it into blk_mq_dispatch_rq_list()? The list is

The problem is that blk_mq_dispatch_rq_list() don't know if it is handling requests from hctx->dispatch or sw/scheduler queue. We only need to clear the bit after hctx->dispatch is flushed. So the clearing can't be moved into blk_mq_dispatch_rq_list().

Quoted 5 lines

generally empty, unless for the case where we splice residuals back. If we splice them back, we grab the lock anyway.

The other places it's set under the hctx lock, yet we end up using an atomic operation to do it.

Quoted 6 lines · Ming Lei

SCSI sets q->queue_depth from shost->cmd_per_lun, and q->queue_depth is per request_queue and more related to scheduler queue compared with hw queue depth, which can be shared by queues, such as TAG_SHARED.

This patch trys to use q->queue_depth as hint for computing

             ^^^^
             tries?

Quoted 2 lines

q->nr_requests, which should be more effective than current way.

Reviewed-by Bart Van Assche <[email protected]>

block/blk-mq.c int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) · line 2677

Quoted 11 lines · Ming Lei

block/blk-mq.c +11 -0

+	/*
+	 * if there is q->queue_depth, all hw queues share
+	 * this queue depth limit
+	 */
+	if (q->queue_depth) {
+		queue_for_each_hw_ctx(q, hctx, i)
+			hctx->flags |= BLK_MQ_F_SHARED_DEPTH;
+	}
+
+	if (!q->elevator)
+		goto exit;

Hello Ming,

It seems very fragile to me to set BLK_MQ_F_SHARED_DEPTH if and only if q->queue_depth != 0. Wouldn't it be better to let the block driver tell the block layer whether or not there is a queue depth limit across hardware queues, e.g. through a tag set flag?

block/blk-mq.c int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) · line 2677

Quoted 11 lines · Ming Lei

block/blk-mq.c +11 -0

+	/*
+	 * if there is q->queue_depth, all hw queues share
+	 * this queue depth limit
+	 */
+	if (q->queue_depth) {
+		queue_for_each_hw_ctx(q, hctx, i)
+			hctx->flags |= BLK_MQ_F_SHARED_DEPTH;
+	}
+
+	if (!q->elevator)
+		goto exit;

Hello Ming,

It seems very fragile to me to set BLK_MQ_F_SHARED_DEPTH if and only if q->queue_depth != 0. Wouldn't it be better to let the block driver tell the block layer whether or not there is a queue depth limit across hardware queues, e.g. through a tag set flag?

Quoted 21 lines · Ming Lei

Quoted 19 lines · Bart Van Assche

block/blk-mq.c int blk_mq_update_nr_requests(struct request_queue *q, unsigned int nr) · line 2677

Quoted 11 lines · Ming Lei

block/blk-mq.c +11 -0

+	/*
+	 * if there is q->queue_depth, all hw queues share
+	 * this queue depth limit
+	 */
+	if (q->queue_depth) {
+		queue_for_each_hw_ctx(q, hctx, i)
+			hctx->flags |= BLK_MQ_F_SHARED_DEPTH;
+	}
+
+	if (!q->elevator)
+		goto exit;

Hello Ming,

It seems very fragile to me to set BLK_MQ_F_SHARED_DEPTH if and only if q->queue_depth != 0. Wouldn't it be better to let the block driver tell the block layer whether or not there is a queue depth limit across hardware queues, e.g. through a tag set flag?

Quoted 2 lines

One reason for not doing in that way is because q->queue_depth can be changed via sysfs interface.

Only the SCSI core allows the queue depth to be changed through sysfs. The other block drivers I am familiar with set the queue depth when the block layer queue is created and do not allow the queue depth to be changed later on.

Quoted 3 lines

Another reason is that better to not exposing this flag to drivers since it isn't necessary, that said it is an internal flag actually.

As far as I know only the SCSI core can create request queues that have a queue depth that is lower than the number of tags in the tag set. So for all block drivers except the SCSI core it is OK to dispatch all requests at once to which a hardware tag has been assigned. My proposal is either to let drivers like the SCSI core set BLK_MQ_F_SHARED_DEPTH or to modify blk_mq_sched_dispatch_requests() such that it flushes all requests if the request queue depth is not lower than the hardware tag set size instead of if q->queue_depth == 0.

Quoted 4 lines · Ming Lei

block/blk-mq.h +4 -0

+static inline bool blk_mq_hctx_is_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	return test_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}

block/blk-mq.h static inline bool blk_mq_hw_queue_mapped(struct blk_mq_hw_ctx *hctx) · line 152

Quoted 10 lines

block/blk-mq.h +10 -0

+
+static inline void blk_mq_hctx_set_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	set_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}
+
+static inline void blk_mq_hctx_clear_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	clear_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}

Hello Ming,

Are these helper functions modified in a later patch? If not, please drop this patch. One line helper functions are not useful and do not improve readability of source code.

Quoted 18 lines · Bart Van Assche

Quoted 4 lines · Ming Lei

block/blk-mq.h +4 -0

+static inline bool blk_mq_hctx_is_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	return test_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}

block/blk-mq.h static inline bool blk_mq_hw_queue_mapped(struct blk_mq_hw_ctx *hctx) · line 152

Quoted 10 lines

block/blk-mq.h +10 -0

+
+static inline void blk_mq_hctx_set_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	set_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}
+
+static inline void blk_mq_hctx_clear_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	clear_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}

Hello Ming,

Quoted 3 lines

Are these helper functions modified in a later patch? If not, please drop this patch. One line helper functions are not useful and do not improve readability of source code.

Agree, they just obfuscate the code. Only reason to do this is to do things like:

static inline void blk_mq_hctx_clear_dispatch_busy(struct blk_mq_hw_ctx *hctx)

{
        if (test_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state))
	        clear_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
}

to avoid unecessary RMW (and locked instruction). At least then you can add a single comment as to why it's done that way. Apart from that, I prefer to open-code it so people don't have to grep to figure out wtf blk_mq_hctx_clear_dispatch_busy() does.

Quoted 4 lines · Ming Lei

block/blk-mq.h +4 -0

+static inline bool blk_mq_hctx_is_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	return test_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}

block/blk-mq.h static inline bool blk_mq_hw_queue_mapped(struct blk_mq_hw_ctx *hctx) · line 152

Quoted 10 lines

block/blk-mq.h +10 -0

+
+static inline void blk_mq_hctx_set_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	set_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}
+
+static inline void blk_mq_hctx_clear_dispatch_busy(struct blk_mq_hw_ctx *hctx)
+{
+	clear_bit(BLK_MQ_S_DISPATCH_BUSY, &hctx->state);
+}

Hello Ming,

Are these helper functions modified in a later patch? If not, please drop this patch. One line helper functions are not useful and do not improve readability of source code.

Quoted 29 lines · Ming Lei

block/blk-mq.h +29 -0

+static inline bool blk_mq_has_dispatch_rqs(struct blk_mq_hw_ctx *hctx)
+{
+	return !list_empty_careful(&hctx->dispatch);
+}
+
+static inline void blk_mq_add_rq_to_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct request *rq)
+{
+	spin_lock(&hctx->lock);
+	list_add(&rq->queuelist, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_add_list_to_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_init(list, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_add_list_to_dispatch_tail(struct blk_mq_hw_ctx *hctx,
+						    struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_tail_init(list, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);

block/blk-mq.h static inline void blk_mq_hctx_clear_dispatch_busy(struct blk_mq_hw_ctx *hctx) · line 192

Quoted 10 lines

block/blk-mq.h +10 -0

+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_take_list_from_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_init(&hctx->dispatch, list);
+	spin_unlock(&hctx->lock);
+}

Same comment for this patch: these helper functions are so short that I'm not sure it is useful to introduce these helper functions.

Bart.

Quoted 44 lines · Bart Van Assche

Quoted 29 lines · Ming Lei

block/blk-mq.h +29 -0

+static inline bool blk_mq_has_dispatch_rqs(struct blk_mq_hw_ctx *hctx)
+{
+	return !list_empty_careful(&hctx->dispatch);
+}
+
+static inline void blk_mq_add_rq_to_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct request *rq)
+{
+	spin_lock(&hctx->lock);
+	list_add(&rq->queuelist, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_add_list_to_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_init(list, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_add_list_to_dispatch_tail(struct blk_mq_hw_ctx *hctx,
+						    struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_tail_init(list, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);

block/blk-mq.h static inline void blk_mq_hctx_clear_dispatch_busy(struct blk_mq_hw_ctx *hctx) · line 192

Quoted 10 lines

block/blk-mq.h +10 -0

+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_take_list_from_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_init(&hctx->dispatch, list);
+	spin_unlock(&hctx->lock);
+}

Same comment for this patch: these helper functions are so short that I'm not sure it is useful to introduce these helper functions.

Quoted 1 line

Bart.

Personally, I like those very much as they give a place to hook up different dispatch_list handling without having to change blk-mq.c and blk-mq-sched.c all over the place.

I am thinking of SMR (zoned block device) support here since we need to to sort insert write requests in blk_mq_add_rq_to_dispatch() and blk_mq_add_list_to_dispatch_tail(). For this last one, the name would

become a little awkward though. This sort insert would be to avoid breaking a sequential write request sequence sent by the disk user. This is needed since this reordering breakage cannot be solved only from the SCSI layer.

Quoted 52 lines · Damien Le Moal

Quoted 44 lines · Bart Van Assche

Quoted 29 lines · Ming Lei

block/blk-mq.h +29 -0

+static inline bool blk_mq_has_dispatch_rqs(struct blk_mq_hw_ctx *hctx)
+{
+	return !list_empty_careful(&hctx->dispatch);
+}
+
+static inline void blk_mq_add_rq_to_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct request *rq)
+{
+	spin_lock(&hctx->lock);
+	list_add(&rq->queuelist, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_add_list_to_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_init(list, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_add_list_to_dispatch_tail(struct blk_mq_hw_ctx *hctx,
+						    struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_tail_init(list, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);

block/blk-mq.h static inline void blk_mq_hctx_clear_dispatch_busy(struct blk_mq_hw_ctx *hctx) · line 192

Quoted 10 lines

block/blk-mq.h +10 -0

+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_take_list_from_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_init(&hctx->dispatch, list);
+	spin_unlock(&hctx->lock);
+}

Same comment for this patch: these helper functions are so short that I'm not sure it is useful to introduce these helper functions.

Quoted 1 line

Bart.

Personally, I like those very much as they give a place to hook up different dispatch_list handling without having to change blk-mq.c and blk-mq-sched.c all over the place.

Quoted 3 lines

I am thinking of SMR (zoned block device) support here since we need to to sort insert write requests in blk_mq_add_rq_to_dispatch() and blk_mq_add_list_to_dispatch_tail(). For this last one, the name would

Could you explain a bit why you sort insert write rq in these two helpers? which are only triggered in case of dispatch busy.

Quoted 5 lines

become a little awkward though. This sort insert would be to avoid breaking a sequential write request sequence sent by the disk user. This is needed since this reordering breakage cannot be solved only from the SCSI layer.

Basically this patchset tries to flush hctx->dispatch first, then fetch requests from scheduler queue after htct->dispatch is flushed.

But it isn't strictly in this way because the implementation is lockless.

So looks the request from hctx->dispatch won't break one coming requests from scheduler.

Quoted 34 lines · Ming Lei

block/blk-mq.h +34 -0

+static inline bool blk_mq_has_dispatch_rqs(struct blk_mq_hw_ctx *hctx)
+{
+	return !list_empty_careful(&hctx->dispatch);
+}
+
+static inline void blk_mq_add_rq_to_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct request *rq)
+{
+	spin_lock(&hctx->lock);
+	list_add(&rq->queuelist, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_add_list_to_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_init(list, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_add_list_to_dispatch_tail(struct blk_mq_hw_ctx *hctx,
+						    struct list_head *list)
+{
+	spin_lock(&hctx->lock);
+	list_splice_tail_init(list, &hctx->dispatch);
+	blk_mq_hctx_set_dispatch_busy(hctx);
+	spin_unlock(&hctx->lock);
+}
+
+static inline void blk_mq_take_list_from_dispatch(struct blk_mq_hw_ctx *hctx,
+		struct list_head *list)

block/blk-mq.h static inline void blk_mq_hctx_clear_dispatch_busy(struct blk_mq_hw_ctx *hctx) · line 192

Quoted 5 lines

block/blk-mq.h +5 -0

+{
+	spin_lock(&hctx->lock);
+	list_splice_init(&hctx->dispatch, list);
+	spin_unlock(&hctx->lock);
+}

Same comment for this patch: these helper functions are so short that I'm not sure it is useful to introduce these helper functions.

Quoted 5 lines · Ming Lei

This patch introduces per-request_queue dispatch list for this purpose, and only when all requests in this list are dispatched out successfully, we can restart to dequeue request from sw/scheduler queue and dispath it to lld.

Wasn't one of the design goals of blk-mq to avoid a per-request_queue list?

In Red Hat internal storage test wrt. blk-mq scheduler, we found that I/O performance is much bad with mq-deadline, especially about sequential I/O on some multi-queue SCSI devcies(lpfc, qla2xxx, SRP...)

Turns out one big issue causes the performance regression: requests are still dequeued from sw queue/scheduler queue even when ldd's queue is busy, so I/O merge becomes quite difficult to make, then sequential IO degrades a lot.

The 1st five patches improve this situation, and brings back some performance loss.

But looks they are still not enough. It is caused by the shared queue depth among all hw queues. For SCSI devices, .cmd_per_lun defines the max number of pending I/O on one request queue, which is per-request_queue depth. So during dispatch, if one hctx is too busy to move on, all hctxs can't dispatch too because of the per-request_queue depth.

Patch 6 ~ 14 use per-request_queue dispatch list to avoid to dequeue requests from sw/scheduler queue when lld queue is busy.

Patch 15 ~20 improve bio merge via hash table in sw queue, which makes bio merge more efficient than current approch in which only the last 8 requests are checked. Since patch 6~14 converts to the scheduler way of dequeuing one request from sw queue one time for SCSI device, and the times of acquring ctx->lock is increased, and merging bio via hash table decreases holding time of ctx->lock and should eliminate effect from patch 14.

With this changes, SCSI-MQ sequential I/O performance is improved much, for lpfc, it is basically brought back compared with block legacy path[1], especially mq-deadline is improved by > X10 [1] on lpfc and by > 3X on SCSI SRP, For mq-none it is improved by 10% on lpfc, and write is improved by > 10% on SRP too.

Also Bart worried that this patchset may affect SRP, so provide test data on SCSI SRP this time:

- fio(libaio, bs:4k, dio, queue_depth:64, 64 jobs)
- system(16 cores, dual sockets, mem: 96G)

               |v4.13-rc3     |v4.13-rc3     | v4.13-rc3+patches |
               |blk-legacy dd |blk-mq none   | blk-mq none  |
-----------------------------------------------------------|
read     :iops|         587K |         526K |         537K |
randread :iops|         115K |         140K |         139K |
write    :iops|         596K |         519K |         602K |
randwrite:iops|         103K |         122K |         120K |

               |v4.13-rc3     |v4.13-rc3     | v4.13-rc3+patches
               |blk-legacy dd |blk-mq dd     | blk-mq dd    |
------------------------------------------------------------
read     :iops|         587K |         155K |         522K |
randread :iops|         115K |         140K |         141K |
write    :iops|         596K |         135K |         587K |
randwrite:iops|         103K |         120K |         118K |

V2:
- dequeue request from sw queues in round roubin's style as suggested by Bart, and introduces one helper in sbitmap for this purpose
- improve bio merge via hash table from sw queue
- add comments about using DISPATCH_BUSY state in lockless way, simplifying handling on busy state,
- hold ctx->lock when clearing ctx busy bit as suggested by Bart

[1] http://marc.info/?l=linux-block&m=150151989915776&w=2

Ming Lei (20):
blk-mq-sched: fix scheduler bad performance
sbitmap: introduce __sbitmap_for_each_set()
blk-mq: introduce blk_mq_dispatch_rq_from_ctx()
blk-mq-sched: move actual dispatching into one helper
blk-mq-sched: improve dispatching from sw queue
blk-mq-sched: don't dequeue request until all in ->dispatch are
flushed
blk-mq-sched: introduce blk_mq_sched_queue_depth()
blk-mq-sched: use q->queue_depth as hint for q->nr_requests
blk-mq: introduce BLK_MQ_F_SHARED_DEPTH
blk-mq-sched: introduce helpers for query, change busy state
blk-mq: introduce helpers for operating ->dispatch list
blk-mq: introduce pointers to dispatch lock & list
blk-mq: pass 'request_queue *' to several helpers of operating BUSY
blk-mq-sched: improve IO scheduling on SCSI devcie
block: introduce rqhash helpers
block: move actual bio merge code into __elv_merge
block: add check on elevator for supporting bio merge via hashtable
from blk-mq sw queue
block: introduce .last_merge and .hash to blk_mq_ctx
blk-mq-sched: refactor blk_mq_sched_try_merge()
blk-mq: improve bio merge from blk-mq sw queue

  block/blk-mq-debugfs.c  |  12 ++--
  block/blk-mq-sched.c    | 187 +++++++++++++++++++++++++++++-------------------
  block/blk-mq-sched.h    |  23 ++++++
  block/blk-mq.c          | 133 +++++++++++++++++++++++++++++++---
  block/blk-mq.h          |  73 +++++++++++++++++++
  block/blk-settings.c    |   2 +
  block/blk.h             |  55 ++++++++++++++
  block/elevator.c        |  93 ++++++++++++++----------
  include/linux/blk-mq.h  |   5 ++
  include/linux/blkdev.h  |   5 ++
  include/linux/sbitmap.h |  54 ++++++++++----
  11 files changed, 504 insertions(+), 138 deletions(-)

Quoted 7 lines · Laurence Oberman

I tested this series using Ming's tests as well as my own set of tests typically run against changes to upstream code in my SRP test-bed. My tests also include very large sequential buffered and un-buffered I/O.

This series seems to be fine for me. I did uncover another issue that is unrelated to these patches and also exists in 4.13-RC3 generic that I am still debugging.

Hello Laurence,

What kind of tests did you run? Only functional tests or also performance tests?

Has the issue you encountered with kernel 4.13-rc3 already been reported on the linux-rdma mailing list?

Thanks,

Bart.

Quoted 14 lines · Bart Van Assche

Quoted 7 lines · Laurence Oberman

I tested this series using Ming's tests as well as my own set of tests typically run against changes to upstream code in my SRP test-bed. My tests also include very large sequential buffered and un-buffered I/O.

This series seems to be fine for me. I did uncover another issue that is unrelated to these patches and also exists in 4.13-RC3 generic that I am still debugging.

Hello Laurence,

What kind of tests did you run? Only functional tests or also performance tests?

Quoted 7 lines

Has the issue you encountered with kernel 4.13-rc3 already been reported on the linux-rdma mailing list?

Thanks,

Bart.

Hi Bart

Actually I was focusing on just performance to see if we had any regressions with Mings new patches for the large sequential I/O cases.

Ming had already tested the small I/O performance cases so I was making sure the large I/O sequential tests did not suffer.

The 4MB un-buffered direct read tests to DM devices seems to perform much the same in my test bed. The 4MB buffered and un-buffered 4MB writes also seem to be well behaved with not much improvement.

These were not exhaustive tests and did not include my usual port disconnect and recovery tests either. I was just making sure we did not regress with Ming's changes.

I was only just starting to baseline test the mq-deadline scheduler as prior to 4.13-RC3 I had not been testing any of the new MQ schedulers. I had always only tested with [none]

The tests were with [none] and [mq-deadline]

The new issue I started seeing was not yet reported yet as I was still investigating it.

In summary:
With buffered writes we see the clone fail in blk_get_request in both Mings kernel and in the upstream 4.13-RC3 stock kernel

[  885.271451] io scheduler mq-deadline registered
[  898.455442] device-mapper: multipath: blk_get_request() returned -11 

- requeuing

This is due to

multipath_clone_and_map()

/*
* Map cloned requests (request-based multipath)
*/
static int multipath_clone_and_map(struct dm_target *ti, struct request *rq,
union map_info *map_context, struct request **__clone)
{
..
..
..

         clone = blk_get_request(q, rq->cmd_flags | REQ_NOMERGE, 
GFP_ATOMIC);
         if (IS_ERR(clone)) {
                 /* EBUSY, ENODEV or EWOULDBLOCK: requeue */
                 bool queue_dying = blk_queue_dying(q);
                 DMERR_LIMIT("blk_get_request() returned %ld%s - 
requeuing",
                             PTR_ERR(clone), queue_dying ? " (path 
offline)" : "");
                 if (queue_dying) {
                         atomic_inc(&m->pg_init_in_progress);
                         activate_or_offline_path(pgpath);
                         return DM_MAPIO_REQUEUE;
                 }
                 return DM_MAPIO_DELAY_REQUEUE;
         }

Still investigating but it leads to a hard lockup

So I still need to see if the hard-lockup happens in the stock kernel with mq-deadline and some other work before coming up with a full summary of the issue.

I also intend to re-run all tests including disconnect and reconnect tests on both mq-deadline and none.

Trace below

[ 1553.167357] NMI watchdog: Watchdog detected hard LOCKUP on cpu 4
[ 1553.167359] Modules linked in: mq_deadline binfmt_misc dm_round_robin xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 rpcrdma ip6table_mangle ip6table_security ip6table_raw iptable_nat ib_isert nf_conntrack_ipv4 iscsi_target_mod nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ib_iser libiscsi scsi_transport_iscsi iptable_mangle iptable_security iptable_raw ebtable_filter ebtables target_core_mod ip6table_filter ip6_tables iptable_filter ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul
[ 1553.167385]  crc32_pclmul ghash_clmulni_intel pcbc aesni_intel sg joydev ipmi_si hpilo crypto_simd hpwdt iTCO_wdt cryptd ipmi_devintf glue_helper gpio_ich iTCO_vendor_support shpchp ipmi_msghandler pcspkr acpi_power_meter i7core_edac lpc_ich pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_multipath ip_tables xfs libcrc32c sd_mod amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm mlxfw i2c_core ptp serio_raw hpsa crc32c_intel bnx2 pps_core devlink scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_srpt]
[ 1553.167410] CPU: 4 PID: 11532 Comm: dd Tainted: G          I 4.13.0-rc3lobeming.ming_V4+ #20
[ 1553.167411] Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
[ 1553.167412] task: ffff9d9344b0d800 task.stack: ffffb5cb913b0000
[ 1553.167421] RIP: 0010:__blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167421] RSP: 0018:ffff9d9377a83d08 EFLAGS: 00000046
[ 1553.167422] RAX: 0000000000001000 RBX: 0000000000001000 RCX: ffff9d91b8e9f500
[ 1553.167423] RDX: fffffcd56af20f00 RSI: ffff9d91b8e9f588 RDI: ffff9d91b8e9f488
[ 1553.167424] RBP: ffff9d9377a83d88 R08: 000000000003c000 R09: 0000000000000001
[ 1553.167424] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1553.167425] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
[ 1553.167426] FS:  00007f3061002740(0000) GS:ffff9d9377a80000(0000) knlGS:0000000000000000
[ 1553.167427] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1553.167428] CR2: 00007f305a511000 CR3: 0000000b1bfb2000 CR4: 00000000000006e0
[ 1553.167429] Call Trace:
[ 1553.167432]  <IRQ>
[ 1553.167437]  ? mempool_free+0x2b/0x80
[ 1553.167440]  blk_recalc_rq_segments+0x28/0x40
[ 1553.167442]  blk_update_request+0x249/0x310
[ 1553.167450]  end_clone_bio+0x46/0x70 [dm_mod]
[ 1553.167453]  bio_endio+0xa1/0x120
[ 1553.167454]  blk_update_request+0xb7/0x310
[ 1553.167457]  scsi_end_request+0x38/0x1d0
[ 1553.167458]  scsi_io_completion+0x13c/0x630
[ 1553.167460]  scsi_finish_command+0xd9/0x120
[ 1553.167462]  scsi_softirq_done+0x12a/0x150
[ 1553.167463]  __blk_mq_complete_request_remote+0x13/0x20
[ 1553.167466]  flush_smp_call_function_queue+0x52/0x110
[ 1553.167468]  generic_smp_call_function_single_interrupt+0x13/0x30
[ 1553.167470]  smp_call_function_single_interrupt+0x27/0x40
[ 1553.167471]  call_function_single_interrupt+0x93/0xa0
[ 1553.167473] RIP: 0010:radix_tree_next_chunk+0xcb/0x2e0
[ 1553.167474] RSP: 0018:ffffb5cb913b3a68 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff04
[ 1553.167475] RAX: 0000000000000001 RBX: 0000000000000010 RCX: ffff9d9e2c556938
[ 1553.167476] RDX: ffff9d93053c5919 RSI: 0000000000000001 RDI: ffff9d9e2c556910
[ 1553.167476] RBP: ffffb5cb913b3ab8 R08: ffffb5cb913b3bc0 R09: 0000000ab7d52000
[ 1553.167477] R10: fffffcd56adf5440 R11: 0000000000000000 R12: 0000000000000230
[ 1553.167477] R13: 0000000000000000 R14: 000000000006e0fe R15: ffff9d9e2c556910
[ 1553.167478]  </IRQ>
[ 1553.167480]  ? radix_tree_next_chunk+0x10b/0x2e0
[ 1553.167481]  find_get_pages_tag+0x149/0x270
[ 1553.167485]  ? block_write_full_page+0xcd/0xe0
[ 1553.167487]  pagevec_lookup_tag+0x21/0x30
[ 1553.167488]  write_cache_pages+0x14c/0x510
[ 1553.167490]  ? __wb_calc_thresh+0x140/0x140
[ 1553.167492]  generic_writepages+0x51/0x80
[ 1553.167493]  blkdev_writepages+0x2f/0x40
[ 1553.167494]  do_writepages+0x1c/0x70
[ 1553.167495]  __filemap_fdatawrite_range+0xc6/0x100
[ 1553.167497]  filemap_write_and_wait+0x3d/0x80
[ 1553.167498]  __sync_blockdev+0x1f/0x40
[ 1553.167499]  __blkdev_put+0x74/0x200
[ 1553.167500]  blkdev_put+0x4c/0xd0
[ 1553.167501]  blkdev_close+0x25/0x30
[ 1553.167503]  __fput+0xe7/0x210
[ 1553.167504]  ____fput+0xe/0x10
[ 1553.167508]  task_work_run+0x83/0xb0
[ 1553.167511]  exit_to_usermode_loop+0x66/0x98
[ 1553.167512]  do_syscall_64+0x13a/0x150
[ 1553.167519]  entry_SYSCALL64_slow_path+0x25/0x25
[ 1553.167520] RIP: 0033:0x7f3060b25bd0
[ 1553.167521] RSP: 002b:00007ffc5e6b12c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[ 1553.167522] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f3060b25bd0
[ 1553.167522] RDX: 0000000000400000 RSI: 0000000000000000 RDI: 0000000000000001
[ 1553.167523] RBP: 0000000000000320 R08: ffffffffffffffff R09: 0000000000402003
[ 1553.167524] R10: 00007ffc5e6b1040 R11: 0000000000000246 R12: 0000000000000320
[ 1553.167525] R13: 0000000000000000 R14: 00007ffc5e6b2618 R15: 00007ffc5e6b1540
[ 1553.167526] Code: db 0f 84 92 01 00 00 45 89 f5 45 89 e2 4c 89 ee 48 c1 e6 04 48 03 71 78 8b 46 08 48 8b 16 44 29 e0 48 89 54 24 30 39 d8 0f 47 c3 <44> 03 56 0c 45 84 db 89 44 24 38 44 89 54 24 3c 0f 85 2a 01 00
[ 1553.167540] Kernel panic - not syncing: Hard LOCKUP
[ 1553.167542] CPU: 4 PID: 11532 Comm: dd Tainted: G          I 4.13.0-rc3lobeming.ming_V4+ #20
[ 1553.167542] Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
[ 1553.167543] Call Trace:
[ 1553.167543]  <NMI>
[ 1553.167545]  dump_stack+0x63/0x87
[ 1553.167548]  panic+0xeb/0x245
[ 1553.167550]  nmi_panic+0x3f/0x40
[ 1553.167553]  watchdog_overflow_callback+0xce/0xf0
[ 1553.167556]  __perf_event_overflow+0x54/0xe0
[ 1553.167559]  perf_event_overflow+0x14/0x20
[ 1553.167562]  intel_pmu_handle_irq+0x203/0x4b0
[ 1553.167566]  perf_event_nmi_handler+0x2d/0x50
[ 1553.167568]  nmi_handle+0x61/0x110
[ 1553.167569]  default_do_nmi+0x44/0x120
[ 1553.167571]  do_nmi+0x113/0x190
[ 1553.167573]  end_repeat_nmi+0x1a/0x1e
[ 1553.167574] RIP: 0010:__blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167575] RSP: 0018:ffff9d9377a83d08 EFLAGS: 00000046
[ 1553.167576] RAX: 0000000000001000 RBX: 0000000000001000 RCX: ffff9d91b8e9f500
[ 1553.167576] RDX: fffffcd56af20f00 RSI: ffff9d91b8e9f588 RDI: ffff9d91b8e9f488
[ 1553.167577] RBP: ffff9d9377a83d88 R08: 000000000003c000 R09: 0000000000000001
[ 1553.167577] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1553.167578] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
[ 1553.167580]  ? __blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167581]  ? __blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167582]  </NMI>
[ 1553.167582]  <IRQ>
[ 1553.167584]  ? mempool_free+0x2b/0x80
[ 1553.167585]  blk_recalc_rq_segments+0x28/0x40
[ 1553.167586]  blk_update_request+0x249/0x310
[ 1553.167590]  end_clone_bio+0x46/0x70 [dm_mod]
[ 1553.167592]  bio_endio+0xa1/0x120
[ 1553.167593]  blk_update_request+0xb7/0x310
[ 1553.167594]  scsi_end_request+0x38/0x1d0
[ 1553.167595]  scsi_io_completion+0x13c/0x630
[ 1553.167597]  scsi_finish_command+0xd9/0x120
[ 1553.167598]  scsi_softirq_done+0x12a/0x150
[ 1553.167599]  __blk_mq_complete_request_remote+0x13/0x20
[ 1553.167601]  flush_smp_call_function_queue+0x52/0x110
[ 1553.167602]  generic_smp_call_function_single_interrupt+0x13/0x30
[ 1553.167603]  smp_call_function_single_interrupt+0x27/0x40
[ 1553.167604]  call_function_single_interrupt+0x93/0xa0
[ 1553.167605] RIP: 0010:radix_tree_next_chunk+0xcb/0x2e0
[ 1553.167606] RSP: 0018:ffffb5cb913b3a68 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff04
[ 1553.167607] RAX: 0000000000000001 RBX: 0000000000000010 RCX: ffff9d9e2c556938
[ 1553.167607] RDX: ffff9d93053c5919 RSI: 0000000000000001 RDI: ffff9d9e2c556910
[ 1553.167608] RBP: ffffb5cb913b3ab8 R08: ffffb5cb913b3bc0 R09: 0000000ab7d52000
[ 1553.167609] R10: fffffcd56adf5440 R11: 0000000000000000 R12: 0000000000000230
[ 1553.167609] R13: 0000000000000000 R14: 000000000006e0fe R15: ffff9d9e2c556910
[ 1553.167610]  </IRQ>
[ 1553.167611]  ? radix_tree_next_chunk+0x10b/0x2e0
[ 1553.167613]  find_get_pages_tag+0x149/0x270
[ 1553.167615]  ? block_write_full_page+0xcd/0xe0
[ 1553.167616]  pagevec_lookup_tag+0x21/0x30
[ 1553.167617]  write_cache_pages+0x14c/0x510
[ 1553.167619]  ? __wb_calc_thresh+0x140/0x140
[ 1553.167621]  generic_writepages+0x51/0x80
[ 1553.167622]  blkdev_writepages+0x2f/0x40
[ 1553.167623]  do_writepages+0x1c/0x70
[ 1553.167627]  __filemap_fdatawrite_range+0xc6/0x100
[ 1553.167629]  filemap_write_and_wait+0x3d/0x80
[ 1553.167630]  __sync_blockdev+0x1f/0x40
[ 1553.167631]  __blkdev_put+0x74/0x200
[ 1553.167632]  blkdev_put+0x4c/0xd0
[ 1553.167633]  blkdev_close+0x25/0x30
[ 1553.167634]  __fput+0xe7/0x210
[ 1553.167635]  ____fput+0xe/0x10
[ 1553.167636]  task_work_run+0x83/0xb0
[ 1553.167637]  exit_to_usermode_loop+0x66/0x98
[ 1553.167638]  do_syscall_64+0x13a/0x150
[ 1553.167640]  entry_SYSCALL64_slow_path+0x25/0x25
[ 1553.167641] RIP: 0033:0x7f3060b25bd0
[ 1553.167641] RSP: 002b:00007ffc5e6b12c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[ 1553.167642] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f3060b25bd0
[ 1553.167643] RDX: 0000000000400000 RSI: 0000000000000000 RDI: 0000000000000001
[ 1553.167643] RBP: 0000000000000320 R08: ffffffffffffffff R09: 0000000000402003
[ 1553.167644] R10: 00007ffc5e6b1040 R11: 0000000000000246 R12: 0000000000000320
[ 1553.167645] R13: 0000000000000000 R14: 00007ffc5e6b2618 R15: 00007ffc5e6b1540
[ 1553.167829] Kernel Offset: 0x2f400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1558.119352] ---[ end Kernel panic - not syncing: Hard LOCKUP
[ 1558.151334] sched: Unexpected reschedule of offline CPU#0!
[ 1558.151342] ------------[ cut here ]------------
[ 1558.151347] WARNING: CPU: 4 PID: 11532 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3c/0x40
[ 1558.151348] Modules linked in: mq_deadline binfmt_misc dm_round_robin xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 rpcrdma ip6table_mangle ip6table_security ip6table_raw iptable_nat ib_isert nf_conntrack_ipv4 iscsi_target_mod nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ib_iser libiscsi scsi_transport_iscsi iptable_mangle iptable_security iptable_raw ebtable_filter ebtables target_core_mod ip6table_filter ip6_tables iptable_filter ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul
[ 1558.151363]  crc32_pclmul ghash_clmulni_intel pcbc aesni_intel sg joydev ipmi_si hpilo crypto_simd hpwdt iTCO_wdt cryptd ipmi_devintf glue_helper gpio_ich iTCO_vendor_support shpchp ipmi_msghandler pcspkr acpi_power_meter i7core_edac lpc_ich pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_multipath ip_tables xfs libcrc32c sd_mod amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm mlxfw i2c_core ptp serio_raw hpsa crc32c_intel bnx2 pps_core devlink scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_srpt]
[ 1558.151377] CPU: 4 PID: 11532 Comm: dd Tainted: G          I 4.13.0-rc3lobeming.ming_V4+ #20
[ 1558.151378] Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
[ 1558.151379] task: ffff9d9344b0d800 task.stack: ffffb5cb913b0000
[ 1558.151380] RIP: 0010:native_smp_send_reschedule+0x3c/0x40
[ 1558.151381] RSP: 0018:ffff9d9377a856a0 EFLAGS: 00010046
[ 1558.151382] RAX: 000000000000002e RBX: 0000000000000000 RCX: 0000000000000006
[ 1558.151383] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9d9377a8e010
[ 1558.151383] RBP: ffff9d9377a856a0 R08: 000000000000002e R09: ffff9d9fa7029552
[ 1558.151384] R10: 0000000000000732 R11: 0000000000000000 R12: ffff9d9377a1bb00
[ 1558.151384] R13: ffff9d936ef70000 R14: ffff9d9377a85758 R15: ffff9d9377a1bb00
[ 1558.151385] FS:  00007f3061002740(0000) GS:ffff9d9377a80000(0000) knlGS:0000000000000000
[ 1558.151386] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1558.151387] CR2: 00007f305a511000 CR3: 0000000b1bfb2000 CR4: 00000000000006e0
[ 1558.151387] Call Trace:
[ 1558.151388]  <NMI>
[ 1558.151391]  resched_curr+0xa1/0xc0
[ 1558.151392]  check_preempt_curr+0x79/0x90
[ 1558.151393]  ttwu_do_wakeup+0x1e/0x160
[ 1558.151395]  ttwu_do_activate+0x7a/0x90
[ 1558.151396]  try_to_wake_up+0x1e1/0x470
[ 1558.151398]  ? vsnprintf+0x1fa/0x4a0
[ 1558.151399]  default_wake_function+0x12/0x20
[ 1558.151403]  __wake_up_common+0x55/0x90
[ 1558.151405]  __wake_up_locked+0x13/0x20
[ 1558.151408]  ep_poll_callback+0xd3/0x2a0
[ 1558.151409]  __wake_up_common+0x55/0x90
[ 1558.151411]  __wake_up+0x39/0x50
[ 1558.151415]  wake_up_klogd_work_func+0x40/0x60
[ 1558.151419]  irq_work_run_list+0x4d/0x70
[ 1558.151421]  ? tick_sched_do_timer+0x70/0x70
[ 1558.151422]  irq_work_tick+0x40/0x50
[ 1558.151424]  update_process_times+0x42/0x60
[ 1558.151426]  tick_sched_handle+0x2d/0x60
[ 1558.151427]  tick_sched_timer+0x39/0x70
[ 1558.151428]  __hrtimer_run_queues+0xe5/0x230
[ 1558.151429]  hrtimer_interrupt+0xa8/0x1a0
[ 1558.151431]  local_apic_timer_interrupt+0x35/0x60
[ 1558.151432]  smp_apic_timer_interrupt+0x38/0x50
[ 1558.151433]  apic_timer_interrupt+0x93/0xa0
[ 1558.151435] RIP: 0010:panic+0x1fa/0x245
[ 1558.151435] RSP: 0018:ffff9d9377a85b30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
[ 1558.151436] RAX: 0000000000000030 RBX: ffff9d890c5d0000 RCX: 0000000000000006
[ 1558.151437] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9d9377a8e010
[ 1558.151438] RBP: ffff9d9377a85ba0 R08: 0000000000000030 R09: ffff9d9fa7029514
[ 1558.151438] R10: 0000000000000731 R11: 0000000000000000 R12: ffffffffb0e63f7f
[ 1558.151439] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1558.151441]  nmi_panic+0x3f/0x40
[ 1558.151442]  watchdog_overflow_callback+0xce/0xf0
[ 1558.151444]  __perf_event_overflow+0x54/0xe0
[ 1558.151445]  perf_event_overflow+0x14/0x20
[ 1558.151446]  intel_pmu_handle_irq+0x203/0x4b0
[ 1558.151449]  perf_event_nmi_handler+0x2d/0x50
[ 1558.151450]  nmi_handle+0x61/0x110
[ 1558.151452]  default_do_nmi+0x44/0x120
[ 1558.151453]  do_nmi+0x113/0x190
[ 1558.151454]  end_repeat_nmi+0x1a/0x1e
[ 1558.151456] RIP: 0010:__blk_recalc_rq_segments+0xec/0x3d0
[ 1558.151456] RSP: 0018:ffff9d9377a83d08 EFLAGS: 00000046
[ 1558.151457] RAX: 0000000000001000 RBX: 0000000000001000 RCX: ffff9d91b8e9f500
[ 1558.151458] RDX: fffffcd56af20f00 RSI: ffff9d91b8e9f588 RDI: ffff9d91b8e9f488
[ 1558.151458] RBP: ffff9d9377a83d88 R08: 000000000003c000 R09: 0000000000000001
[ 1558.151459] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1558.151459] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
[ 1558.151461]  ? __blk_recalc_rq_segments+0xec/0x3d0
[ 1558.151463]  ? __blk_recalc_rq_segments+0xec/0x3d0
[ 1558.151463]  </NMI>
[ 1558.151464]  <IRQ>
[ 1558.151465]  ? mempool_free+0x2b/0x80
[ 1558.151467]  blk_recalc_rq_segments+0x28/0x40
[ 1558.151468]  blk_update_request+0x249/0x310
[ 1558.151472]  end_clone_bio+0x46/0x70 [dm_mod]
[ 1558.151473]  bio_endio+0xa1/0x120
[ 1558.151474]  blk_update_request+0xb7/0x310
[ 1558.151476]  scsi_end_request+0x38/0x1d0
[ 1558.151477]  scsi_io_completion+0x13c/0x630
[ 1558.151478]  scsi_finish_command+0xd9/0x120
[ 1558.151480]  scsi_softirq_done+0x12a/0x150
[ 1558.151481]  __blk_mq_complete_request_remote+0x13/0x20
[ 1558.151483]  flush_smp_call_function_queue+0x52/0x110
[ 1558.151484]  generic_smp_call_function_single_interrupt+0x13/0x30
[ 1558.151485]  smp_call_function_single_interrupt+0x27/0x40
[ 1558.151486]  call_function_single_interrupt+0x93/0xa0
[ 1558.151487] RIP: 0010:radix_tree_next_chunk+0xcb/0x2e0
[ 1558.151488] RSP: 0018:ffffb5cb913b3a68 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff04
[ 1558.151489] RAX: 0000000000000001 RBX: 0000000000000010 RCX: ffff9d9e2c556938
[ 1558.151490] RDX: ffff9d93053c5919 RSI: 0000000000000001 RDI: ffff9d9e2c556910
[ 1558.151490] RBP: ffffb5cb913b3ab8 R08: ffffb5cb913b3bc0 R09: 0000000ab7d52000
[ 1558.151491] R10: fffffcd56adf5440 R11: 0000000000000000 R12: 0000000000000230
[ 1558.151492] R13: 0000000000000000 R14: 000000000006e0fe R15: ffff9d9e2c556910
[ 1558.151492]  </IRQ>
[ 1558.151494]  ? radix_tree_next_chunk+0x10b/0x2e0
[ 1558.151496]  find_get_pages_tag+0x149/0x270
[ 1558.151497]  ? block_write_full_page+0xcd/0xe0
[ 1558.151499]  pagevec_lookup_tag+0x21/0x30
[ 1558.151500]  write_cache_pages+0x14c/0x510
[ 1558.151502]  ? __wb_calc_thresh+0x140/0x140
[ 1558.151504]  generic_writepages+0x51/0x80
[ 1558.151505]  blkdev_writepages+0x2f/0x40
[ 1558.151506]  do_writepages+0x1c/0x70
[ 1558.151507]  __filemap_fdatawrite_range+0xc6/0x100
[ 1558.151509]  filemap_write_and_wait+0x3d/0x80
[ 1558.151510]  __sync_blockdev+0x1f/0x40
[ 1558.151511]  __blkdev_put+0x74/0x200
[ 1558.151512]  blkdev_put+0x4c/0xd0
[ 1558.151513]  blkdev_close+0x25/0x30
[ 1558.151514]  __fput+0xe7/0x210
[ 1558.151515]  ____fput+0xe/0x10
[ 1558.151517]  task_work_run+0x83/0xb0
[ 1558.151518]  exit_to_usermode_loop+0x66/0x98
[ 1558.151519]  do_syscall_64+0x13a/0x150
[ 1558.151521]  entry_SYSCALL64_slow_path+0x25/0x25
[ 1558.151522] RIP: 0033:0x7f3060b25bd0

[ 1558.151522] RSP: 002b:00007ffc5e6b12c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[ 1558.151523] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f3060b25bd0
[ 1558.151524] RDX: 0000000000400000 RSI: 0000000000000000 RDI: 0000000000000001
[ 1558.151524] RBP: 0000000000000320 R08: ffffffffffffffff R09: 0000000000402003
[ 1558.151525] R10: 00007ffc5e6b1040 R11: 0000000000000246 R12: 0000000000000320
[ 1558.151526] R13: 0000000000000000 R14: 00007ffc5e6b2618 R15: 00007ffc5e6b1540
[ 1558.151527] Code: dc 00 0f 92 c0 84 c0 74 14 48 8b 05 ff 9d a9 00 be fd 00 00 00 ff 90 a0 00 00 00 5d c3 89 fe 48 c7 c7 90 4d e3 b0 e8 c7 db 09 00 <0f> ff 5d c3 66 66 66 66 90 55 48 89 e5 48 83 ec 20 65 48 8b 04
[ 1558.151540] ---[ end trace 408c28f8c132530c ]---

Quoted 360 lines · Laurence Oberman

Quoted 14 lines · Bart Van Assche

Quoted 7 lines · Laurence Oberman

I tested this series using Ming's tests as well as my own set of tests typically run against changes to upstream code in my SRP test-bed. My tests also include very large sequential buffered and un-buffered I/O.

This series seems to be fine for me. I did uncover another issue that is unrelated to these patches and also exists in 4.13-RC3 generic that I am still debugging.

Hello Laurence,

What kind of tests did you run? Only functional tests or also performance tests?

Quoted 7 lines

Has the issue you encountered with kernel 4.13-rc3 already been reported on the linux-rdma mailing list?

Thanks,

Bart.

Hi Bart

Actually I was focusing on just performance to see if we had any regressions with Mings new patches for the large sequential I/O cases.

Ming had already tested the small I/O performance cases so I was making sure the large I/O sequential tests did not suffer.

The 4MB un-buffered direct read tests to DM devices seems to perform much the same in my test bed. The 4MB buffered and un-buffered 4MB writes also seem to be well behaved with not much improvement.

These were not exhaustive tests and did not include my usual port disconnect and recovery tests either. I was just making sure we did not regress with Ming's changes.

I was only just starting to baseline test the mq-deadline scheduler as prior to 4.13-RC3 I had not been testing any of the new MQ schedulers. I had always only tested with [none]

The tests were with [none] and [mq-deadline]

The new issue I started seeing was not yet reported yet as I was still investigating it.

In summary:
With buffered writes we see the clone fail in blk_get_request in both Mings kernel and in the upstream 4.13-RC3 stock kernel

[  885.271451] io scheduler mq-deadline registered
[  898.455442] device-mapper: multipath: blk_get_request() returned 

-11 - requeuing

This is due to

multipath_clone_and_map()

/*
* Map cloned requests (request-based multipath)
*/
static int multipath_clone_and_map(struct dm_target *ti, struct request *rq,
union map_info *map_context, struct request **__clone)
{
..
..
..

         clone = blk_get_request(q, rq->cmd_flags | REQ_NOMERGE, 
GFP_ATOMIC);
         if (IS_ERR(clone)) {
                 /* EBUSY, ENODEV or EWOULDBLOCK: requeue */
                 bool queue_dying = blk_queue_dying(q);
                 DMERR_LIMIT("blk_get_request() returned %ld%s - 
requeuing",
                             PTR_ERR(clone), queue_dying ? " (path 
offline)" : "");
                 if (queue_dying) {
                         atomic_inc(&m->pg_init_in_progress);
                         activate_or_offline_path(pgpath);
                         return DM_MAPIO_REQUEUE;
                 }
                 return DM_MAPIO_DELAY_REQUEUE;
         }

Still investigating but it leads to a hard lockup

So I still need to see if the hard-lockup happens in the stock kernel with mq-deadline and some other work before coming up with a full summary of the issue.

I also intend to re-run all tests including disconnect and reconnect tests on both mq-deadline and none.

Trace below

[ 1553.167357] NMI watchdog: Watchdog detected hard LOCKUP on cpu 4
[ 1553.167359] Modules linked in: mq_deadline binfmt_misc dm_round_robin xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 rpcrdma ip6table_mangle ip6table_security ip6table_raw iptable_nat ib_isert nf_conntrack_ipv4 iscsi_target_mod nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ib_iser libiscsi scsi_transport_iscsi iptable_mangle iptable_security iptable_raw ebtable_filter ebtables target_core_mod ip6table_filter ip6_tables iptable_filter ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul
[ 1553.167385]  crc32_pclmul ghash_clmulni_intel pcbc aesni_intel sg joydev ipmi_si hpilo crypto_simd hpwdt iTCO_wdt cryptd ipmi_devintf glue_helper gpio_ich iTCO_vendor_support shpchp ipmi_msghandler pcspkr acpi_power_meter i7core_edac lpc_ich pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_multipath ip_tables xfs libcrc32c sd_mod amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm mlxfw i2c_core ptp serio_raw hpsa crc32c_intel bnx2 pps_core devlink scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_srpt]
[ 1553.167410] CPU: 4 PID: 11532 Comm: dd Tainted: G          I 4.13.0-rc3lobeming.ming_V4+ #20
[ 1553.167411] Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
[ 1553.167412] task: ffff9d9344b0d800 task.stack: ffffb5cb913b0000
[ 1553.167421] RIP: 0010:__blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167421] RSP: 0018:ffff9d9377a83d08 EFLAGS: 00000046
[ 1553.167422] RAX: 0000000000001000 RBX: 0000000000001000 RCX: ffff9d91b8e9f500
[ 1553.167423] RDX: fffffcd56af20f00 RSI: ffff9d91b8e9f588 RDI: ffff9d91b8e9f488
[ 1553.167424] RBP: ffff9d9377a83d88 R08: 000000000003c000 R09: 0000000000000001
[ 1553.167424] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1553.167425] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
[ 1553.167426] FS:  00007f3061002740(0000) GS:ffff9d9377a80000(0000) knlGS:0000000000000000
[ 1553.167427] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1553.167428] CR2: 00007f305a511000 CR3: 0000000b1bfb2000 CR4: 00000000000006e0
[ 1553.167429] Call Trace:
[ 1553.167432]  <IRQ>
[ 1553.167437]  ? mempool_free+0x2b/0x80
[ 1553.167440]  blk_recalc_rq_segments+0x28/0x40
[ 1553.167442]  blk_update_request+0x249/0x310
[ 1553.167450]  end_clone_bio+0x46/0x70 [dm_mod]
[ 1553.167453]  bio_endio+0xa1/0x120
[ 1553.167454]  blk_update_request+0xb7/0x310
[ 1553.167457]  scsi_end_request+0x38/0x1d0
[ 1553.167458]  scsi_io_completion+0x13c/0x630
[ 1553.167460]  scsi_finish_command+0xd9/0x120
[ 1553.167462]  scsi_softirq_done+0x12a/0x150
[ 1553.167463]  __blk_mq_complete_request_remote+0x13/0x20
[ 1553.167466]  flush_smp_call_function_queue+0x52/0x110
[ 1553.167468]  generic_smp_call_function_single_interrupt+0x13/0x30
[ 1553.167470]  smp_call_function_single_interrupt+0x27/0x40
[ 1553.167471]  call_function_single_interrupt+0x93/0xa0
[ 1553.167473] RIP: 0010:radix_tree_next_chunk+0xcb/0x2e0
[ 1553.167474] RSP: 0018:ffffb5cb913b3a68 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff04
[ 1553.167475] RAX: 0000000000000001 RBX: 0000000000000010 RCX: ffff9d9e2c556938
[ 1553.167476] RDX: ffff9d93053c5919 RSI: 0000000000000001 RDI: ffff9d9e2c556910
[ 1553.167476] RBP: ffffb5cb913b3ab8 R08: ffffb5cb913b3bc0 R09: 0000000ab7d52000
[ 1553.167477] R10: fffffcd56adf5440 R11: 0000000000000000 R12: 0000000000000230
[ 1553.167477] R13: 0000000000000000 R14: 000000000006e0fe R15: ffff9d9e2c556910
[ 1553.167478]  </IRQ>
[ 1553.167480]  ? radix_tree_next_chunk+0x10b/0x2e0
[ 1553.167481]  find_get_pages_tag+0x149/0x270
[ 1553.167485]  ? block_write_full_page+0xcd/0xe0
[ 1553.167487]  pagevec_lookup_tag+0x21/0x30
[ 1553.167488]  write_cache_pages+0x14c/0x510
[ 1553.167490]  ? __wb_calc_thresh+0x140/0x140
[ 1553.167492]  generic_writepages+0x51/0x80
[ 1553.167493]  blkdev_writepages+0x2f/0x40
[ 1553.167494]  do_writepages+0x1c/0x70
[ 1553.167495]  __filemap_fdatawrite_range+0xc6/0x100
[ 1553.167497]  filemap_write_and_wait+0x3d/0x80
[ 1553.167498]  __sync_blockdev+0x1f/0x40
[ 1553.167499]  __blkdev_put+0x74/0x200
[ 1553.167500]  blkdev_put+0x4c/0xd0
[ 1553.167501]  blkdev_close+0x25/0x30
[ 1553.167503]  __fput+0xe7/0x210
[ 1553.167504]  ____fput+0xe/0x10
[ 1553.167508]  task_work_run+0x83/0xb0
[ 1553.167511]  exit_to_usermode_loop+0x66/0x98
[ 1553.167512]  do_syscall_64+0x13a/0x150
[ 1553.167519]  entry_SYSCALL64_slow_path+0x25/0x25
[ 1553.167520] RIP: 0033:0x7f3060b25bd0
[ 1553.167521] RSP: 002b:00007ffc5e6b12c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[ 1553.167522] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f3060b25bd0
[ 1553.167522] RDX: 0000000000400000 RSI: 0000000000000000 RDI: 0000000000000001
[ 1553.167523] RBP: 0000000000000320 R08: ffffffffffffffff R09: 0000000000402003
[ 1553.167524] R10: 00007ffc5e6b1040 R11: 0000000000000246 R12: 0000000000000320
[ 1553.167525] R13: 0000000000000000 R14: 00007ffc5e6b2618 R15: 00007ffc5e6b1540
[ 1553.167526] Code: db 0f 84 92 01 00 00 45 89 f5 45 89 e2 4c 89 ee 48 c1 e6 04 48 03 71 78 8b 46 08 48 8b 16 44 29 e0 48 89 54 24 30 39 d8 0f 47 c3 <44> 03 56 0c 45 84 db 89 44 24 38 44 89 54 24 3c 0f 85 2a 01 00
[ 1553.167540] Kernel panic - not syncing: Hard LOCKUP
[ 1553.167542] CPU: 4 PID: 11532 Comm: dd Tainted: G          I 4.13.0-rc3lobeming.ming_V4+ #20
[ 1553.167542] Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
[ 1553.167543] Call Trace:
[ 1553.167543]  <NMI>
[ 1553.167545]  dump_stack+0x63/0x87
[ 1553.167548]  panic+0xeb/0x245
[ 1553.167550]  nmi_panic+0x3f/0x40
[ 1553.167553]  watchdog_overflow_callback+0xce/0xf0
[ 1553.167556]  __perf_event_overflow+0x54/0xe0
[ 1553.167559]  perf_event_overflow+0x14/0x20
[ 1553.167562]  intel_pmu_handle_irq+0x203/0x4b0
[ 1553.167566]  perf_event_nmi_handler+0x2d/0x50
[ 1553.167568]  nmi_handle+0x61/0x110
[ 1553.167569]  default_do_nmi+0x44/0x120
[ 1553.167571]  do_nmi+0x113/0x190
[ 1553.167573]  end_repeat_nmi+0x1a/0x1e
[ 1553.167574] RIP: 0010:__blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167575] RSP: 0018:ffff9d9377a83d08 EFLAGS: 00000046
[ 1553.167576] RAX: 0000000000001000 RBX: 0000000000001000 RCX: ffff9d91b8e9f500
[ 1553.167576] RDX: fffffcd56af20f00 RSI: ffff9d91b8e9f588 RDI: ffff9d91b8e9f488
[ 1553.167577] RBP: ffff9d9377a83d88 R08: 000000000003c000 R09: 0000000000000001
[ 1553.167577] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1553.167578] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
[ 1553.167580]  ? __blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167581]  ? __blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167582]  </NMI>
[ 1553.167582]  <IRQ>
[ 1553.167584]  ? mempool_free+0x2b/0x80
[ 1553.167585]  blk_recalc_rq_segments+0x28/0x40
[ 1553.167586]  blk_update_request+0x249/0x310
[ 1553.167590]  end_clone_bio+0x46/0x70 [dm_mod]
[ 1553.167592]  bio_endio+0xa1/0x120
[ 1553.167593]  blk_update_request+0xb7/0x310
[ 1553.167594]  scsi_end_request+0x38/0x1d0
[ 1553.167595]  scsi_io_completion+0x13c/0x630
[ 1553.167597]  scsi_finish_command+0xd9/0x120
[ 1553.167598]  scsi_softirq_done+0x12a/0x150
[ 1553.167599]  __blk_mq_complete_request_remote+0x13/0x20
[ 1553.167601]  flush_smp_call_function_queue+0x52/0x110
[ 1553.167602]  generic_smp_call_function_single_interrupt+0x13/0x30
[ 1553.167603]  smp_call_function_single_interrupt+0x27/0x40
[ 1553.167604]  call_function_single_interrupt+0x93/0xa0
[ 1553.167605] RIP: 0010:radix_tree_next_chunk+0xcb/0x2e0
[ 1553.167606] RSP: 0018:ffffb5cb913b3a68 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff04
[ 1553.167607] RAX: 0000000000000001 RBX: 0000000000000010 RCX: ffff9d9e2c556938
[ 1553.167607] RDX: ffff9d93053c5919 RSI: 0000000000000001 RDI: ffff9d9e2c556910
[ 1553.167608] RBP: ffffb5cb913b3ab8 R08: ffffb5cb913b3bc0 R09: 0000000ab7d52000
[ 1553.167609] R10: fffffcd56adf5440 R11: 0000000000000000 R12: 0000000000000230
[ 1553.167609] R13: 0000000000000000 R14: 000000000006e0fe R15: ffff9d9e2c556910
[ 1553.167610]  </IRQ>
[ 1553.167611]  ? radix_tree_next_chunk+0x10b/0x2e0
[ 1553.167613]  find_get_pages_tag+0x149/0x270
[ 1553.167615]  ? block_write_full_page+0xcd/0xe0
[ 1553.167616]  pagevec_lookup_tag+0x21/0x30
[ 1553.167617]  write_cache_pages+0x14c/0x510
[ 1553.167619]  ? __wb_calc_thresh+0x140/0x140
[ 1553.167621]  generic_writepages+0x51/0x80
[ 1553.167622]  blkdev_writepages+0x2f/0x40
[ 1553.167623]  do_writepages+0x1c/0x70
[ 1553.167627]  __filemap_fdatawrite_range+0xc6/0x100
[ 1553.167629]  filemap_write_and_wait+0x3d/0x80
[ 1553.167630]  __sync_blockdev+0x1f/0x40
[ 1553.167631]  __blkdev_put+0x74/0x200
[ 1553.167632]  blkdev_put+0x4c/0xd0
[ 1553.167633]  blkdev_close+0x25/0x30
[ 1553.167634]  __fput+0xe7/0x210
[ 1553.167635]  ____fput+0xe/0x10
[ 1553.167636]  task_work_run+0x83/0xb0
[ 1553.167637]  exit_to_usermode_loop+0x66/0x98
[ 1553.167638]  do_syscall_64+0x13a/0x150
[ 1553.167640]  entry_SYSCALL64_slow_path+0x25/0x25
[ 1553.167641] RIP: 0033:0x7f3060b25bd0
[ 1553.167641] RSP: 002b:00007ffc5e6b12c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[ 1553.167642] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f3060b25bd0
[ 1553.167643] RDX: 0000000000400000 RSI: 0000000000000000 RDI: 0000000000000001
[ 1553.167643] RBP: 0000000000000320 R08: ffffffffffffffff R09: 0000000000402003
[ 1553.167644] R10: 00007ffc5e6b1040 R11: 0000000000000246 R12: 0000000000000320
[ 1553.167645] R13: 0000000000000000 R14: 00007ffc5e6b2618 R15: 00007ffc5e6b1540
[ 1553.167829] Kernel Offset: 0x2f400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1558.119352] ---[ end Kernel panic - not syncing: Hard LOCKUP
[ 1558.151334] sched: Unexpected reschedule of offline CPU#0!
[ 1558.151342] ------------[ cut here ]------------
[ 1558.151347] WARNING: CPU: 4 PID: 11532 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3c/0x40
[ 1558.151348] Modules linked in: mq_deadline binfmt_misc dm_round_robin xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 rpcrdma ip6table_mangle ip6table_security ip6table_raw iptable_nat ib_isert nf_conntrack_ipv4 iscsi_target_mod nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ib_iser libiscsi scsi_transport_iscsi iptable_mangle iptable_security iptable_raw ebtable_filter ebtables target_core_mod ip6table_filter ip6_tables iptable_filter ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul
[ 1558.151363]  crc32_pclmul ghash_clmulni_intel pcbc aesni_intel sg joydev ipmi_si hpilo crypto_simd hpwdt iTCO_wdt cryptd ipmi_devintf glue_helper gpio_ich iTCO_vendor_support shpchp ipmi_msghandler pcspkr acpi_power_meter i7core_edac lpc_ich pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_multipath ip_tables xfs libcrc32c sd_mod amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm mlxfw i2c_core ptp serio_raw hpsa crc32c_intel bnx2 pps_core devlink scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_srpt]
[ 1558.151377] CPU: 4 PID: 11532 Comm: dd Tainted: G          I 4.13.0-rc3lobeming.ming_V4+ #20
[ 1558.151378] Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
[ 1558.151379] task: ffff9d9344b0d800 task.stack: ffffb5cb913b0000
[ 1558.151380] RIP: 0010:native_smp_send_reschedule+0x3c/0x40
[ 1558.151381] RSP: 0018:ffff9d9377a856a0 EFLAGS: 00010046
[ 1558.151382] RAX: 000000000000002e RBX: 0000000000000000 RCX: 0000000000000006
[ 1558.151383] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9d9377a8e010
[ 1558.151383] RBP: ffff9d9377a856a0 R08: 000000000000002e R09: ffff9d9fa7029552
[ 1558.151384] R10: 0000000000000732 R11: 0000000000000000 R12: ffff9d9377a1bb00
[ 1558.151384] R13: ffff9d936ef70000 R14: ffff9d9377a85758 R15: ffff9d9377a1bb00
[ 1558.151385] FS:  00007f3061002740(0000) GS:ffff9d9377a80000(0000) knlGS:0000000000000000
[ 1558.151386] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1558.151387] CR2: 00007f305a511000 CR3: 0000000b1bfb2000 CR4: 00000000000006e0
[ 1558.151387] Call Trace:
[ 1558.151388]  <NMI>
[ 1558.151391]  resched_curr+0xa1/0xc0
[ 1558.151392]  check_preempt_curr+0x79/0x90
[ 1558.151393]  ttwu_do_wakeup+0x1e/0x160
[ 1558.151395]  ttwu_do_activate+0x7a/0x90
[ 1558.151396]  try_to_wake_up+0x1e1/0x470
[ 1558.151398]  ? vsnprintf+0x1fa/0x4a0
[ 1558.151399]  default_wake_function+0x12/0x20
[ 1558.151403]  __wake_up_common+0x55/0x90
[ 1558.151405]  __wake_up_locked+0x13/0x20
[ 1558.151408]  ep_poll_callback+0xd3/0x2a0
[ 1558.151409]  __wake_up_common+0x55/0x90
[ 1558.151411]  __wake_up+0x39/0x50
[ 1558.151415]  wake_up_klogd_work_func+0x40/0x60
[ 1558.151419]  irq_work_run_list+0x4d/0x70
[ 1558.151421]  ? tick_sched_do_timer+0x70/0x70
[ 1558.151422]  irq_work_tick+0x40/0x50
[ 1558.151424]  update_process_times+0x42/0x60
[ 1558.151426]  tick_sched_handle+0x2d/0x60
[ 1558.151427]  tick_sched_timer+0x39/0x70
[ 1558.151428]  __hrtimer_run_queues+0xe5/0x230
[ 1558.151429]  hrtimer_interrupt+0xa8/0x1a0
[ 1558.151431]  local_apic_timer_interrupt+0x35/0x60
[ 1558.151432]  smp_apic_timer_interrupt+0x38/0x50
[ 1558.151433]  apic_timer_interrupt+0x93/0xa0
[ 1558.151435] RIP: 0010:panic+0x1fa/0x245
[ 1558.151435] RSP: 0018:ffff9d9377a85b30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
[ 1558.151436] RAX: 0000000000000030 RBX: ffff9d890c5d0000 RCX: 0000000000000006
[ 1558.151437] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9d9377a8e010
[ 1558.151438] RBP: ffff9d9377a85ba0 R08: 0000000000000030 R09: ffff9d9fa7029514
[ 1558.151438] R10: 0000000000000731 R11: 0000000000000000 R12: ffffffffb0e63f7f
[ 1558.151439] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1558.151441]  nmi_panic+0x3f/0x40
[ 1558.151442]  watchdog_overflow_callback+0xce/0xf0
[ 1558.151444]  __perf_event_overflow+0x54/0xe0
[ 1558.151445]  perf_event_overflow+0x14/0x20
[ 1558.151446]  intel_pmu_handle_irq+0x203/0x4b0
[ 1558.151449]  perf_event_nmi_handler+0x2d/0x50
[ 1558.151450]  nmi_handle+0x61/0x110
[ 1558.151452]  default_do_nmi+0x44/0x120
[ 1558.151453]  do_nmi+0x113/0x190
[ 1558.151454]  end_repeat_nmi+0x1a/0x1e
[ 1558.151456] RIP: 0010:__blk_recalc_rq_segments+0xec/0x3d0
[ 1558.151456] RSP: 0018:ffff9d9377a83d08 EFLAGS: 00000046
[ 1558.151457] RAX: 0000000000001000 RBX: 0000000000001000 RCX: ffff9d91b8e9f500
[ 1558.151458] RDX: fffffcd56af20f00 RSI: ffff9d91b8e9f588 RDI: ffff9d91b8e9f488
[ 1558.151458] RBP: ffff9d9377a83d88 R08: 000000000003c000 R09: 0000000000000001
[ 1558.151459] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1558.151459] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
[ 1558.151461]  ? __blk_recalc_rq_segments+0xec/0x3d0
[ 1558.151463]  ? __blk_recalc_rq_segments+0xec/0x3d0
[ 1558.151463]  </NMI>
[ 1558.151464]  <IRQ>
[ 1558.151465]  ? mempool_free+0x2b/0x80
[ 1558.151467]  blk_recalc_rq_segments+0x28/0x40
[ 1558.151468]  blk_update_request+0x249/0x310
[ 1558.151472]  end_clone_bio+0x46/0x70 [dm_mod]
[ 1558.151473]  bio_endio+0xa1/0x120
[ 1558.151474]  blk_update_request+0xb7/0x310
[ 1558.151476]  scsi_end_request+0x38/0x1d0
[ 1558.151477]  scsi_io_completion+0x13c/0x630
[ 1558.151478]  scsi_finish_command+0xd9/0x120
[ 1558.151480]  scsi_softirq_done+0x12a/0x150
[ 1558.151481]  __blk_mq_complete_request_remote+0x13/0x20
[ 1558.151483]  flush_smp_call_function_queue+0x52/0x110
[ 1558.151484]  generic_smp_call_function_single_interrupt+0x13/0x30
[ 1558.151485]  smp_call_function_single_interrupt+0x27/0x40
[ 1558.151486]  call_function_single_interrupt+0x93/0xa0
[ 1558.151487] RIP: 0010:radix_tree_next_chunk+0xcb/0x2e0
[ 1558.151488] RSP: 0018:ffffb5cb913b3a68 EFLAGS: 00000203 ORIG_RAX: ffffffffffffff04
[ 1558.151489] RAX: 0000000000000001 RBX: 0000000000000010 RCX: ffff9d9e2c556938
[ 1558.151490] RDX: ffff9d93053c5919 RSI: 0000000000000001 RDI: ffff9d9e2c556910
[ 1558.151490] RBP: ffffb5cb913b3ab8 R08: ffffb5cb913b3bc0 R09: 0000000ab7d52000
[ 1558.151491] R10: fffffcd56adf5440 R11: 0000000000000000 R12: 0000000000000230
[ 1558.151492] R13: 0000000000000000 R14: 000000000006e0fe R15: ffff9d9e2c556910
[ 1558.151492]  </IRQ>
[ 1558.151494]  ? radix_tree_next_chunk+0x10b/0x2e0
[ 1558.151496]  find_get_pages_tag+0x149/0x270
[ 1558.151497]  ? block_write_full_page+0xcd/0xe0
[ 1558.151499]  pagevec_lookup_tag+0x21/0x30
[ 1558.151500]  write_cache_pages+0x14c/0x510
[ 1558.151502]  ? __wb_calc_thresh+0x140/0x140
[ 1558.151504]  generic_writepages+0x51/0x80
[ 1558.151505]  blkdev_writepages+0x2f/0x40
[ 1558.151506]  do_writepages+0x1c/0x70
[ 1558.151507]  __filemap_fdatawrite_range+0xc6/0x100
[ 1558.151509]  filemap_write_and_wait+0x3d/0x80
[ 1558.151510]  __sync_blockdev+0x1f/0x40
[ 1558.151511]  __blkdev_put+0x74/0x200
[ 1558.151512]  blkdev_put+0x4c/0xd0
[ 1558.151513]  blkdev_close+0x25/0x30
[ 1558.151514]  __fput+0xe7/0x210
[ 1558.151515]  ____fput+0xe/0x10
[ 1558.151517]  task_work_run+0x83/0xb0
[ 1558.151518]  exit_to_usermode_loop+0x66/0x98
[ 1558.151519]  do_syscall_64+0x13a/0x150
[ 1558.151521]  entry_SYSCALL64_slow_path+0x25/0x25
[ 1558.151522] RIP: 0033:0x7f3060b25bd0

Quoted 8 lines

[ 1558.151522] RSP: 002b:00007ffc5e6b12c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
[ 1558.151523] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f3060b25bd0
[ 1558.151524] RDX: 0000000000400000 RSI: 0000000000000000 RDI: 0000000000000001
[ 1558.151524] RBP: 0000000000000320 R08: ffffffffffffffff R09: 0000000000402003
[ 1558.151525] R10: 00007ffc5e6b1040 R11: 0000000000000246 R12: 0000000000000320
[ 1558.151526] R13: 0000000000000000 R14: 00007ffc5e6b2618 R15: 00007ffc5e6b1540
[ 1558.151527] Code: dc 00 0f 92 c0 84 c0 74 14 48 8b 05 ff 9d a9 00 be fd 00 00 00 ff 90 a0 00 00 00 5d c3 89 fe 48 c7 c7 90 4d e3 b0 e8 c7 db 09 00 <0f> ff 5d c3 66 66 66 66 90 55 48 89 e5 48 83 ec 20 65 48 8b 04
[ 1558.151540] ---[ end trace 408c28f8c132530c ]---

Replying to my own email:

Hello Bart

So with the stock 4.13-RC3 using [none] I get the failing clones but no hard lockup.

[  526.677611] multipath_clone_and_map: 148 callbacks suppressed
[  526.707429] device-mapper: multipath: blk_get_request() returned -11 
- requeuing
[  527.283432] device-mapper: multipath: blk_get_request() returned -11 
- requeuing
[  528.324566] device-mapper: multipath: blk_get_request() returned -11 
- requeuing
[  528.367118] device-mapper: multipath: blk_get_request() returned -11 

- requeuing

Then on the same kernel
modprobe mq-deadline and running with [mq-deadline] on stock upstream I DO not get the clone failures and I don't seem to get hard lockups.

I will reach out to Ming once I confirm going back to his kernel gets me back into hard lockups again. I will also mention to him that we see the clone failure when using his kernel with [mq-deadline]

Perhaps his enhancements are indeed contributing

Thanks
Laurence

I also intend to test qla2xxx and mq-deadline shortly

Quoted 14 lines · Bart Van Assche

Quoted 7 lines · Laurence Oberman

I tested this series using Ming's tests as well as my own set of tests typically run against changes to upstream code in my SRP test-bed. My tests also include very large sequential buffered and un-buffered I/O.

This series seems to be fine for me. I did uncover another issue that is unrelated to these patches and also exists in 4.13-RC3 generic that I am still debugging.

Hello Laurence,

What kind of tests did you run? Only functional tests or also performance tests?

Quoted 7 lines

Has the issue you encountered with kernel 4.13-rc3 already been reported on the linux-rdma mailing list?

Thanks,

Bart.

Hi Bart

Actually I was focusing on just performance to see if we had any regressions with Mings new patches for the large sequential I/O cases.

Ming had already tested the small I/O performance cases so I was making sure the large I/O sequential tests did not suffer.

The 4MB un-buffered direct read tests to DM devices seems to perform much the same in my test bed.
The 4MB buffered and un-buffered 4MB writes also seem to be well behaved with not much improvement.

These were not exhaustive tests and did not include my usual port disconnect and recovery tests either.
I was just making sure we did not regress with Ming's changes.

I was only just starting to baseline test the mq-deadline scheduler as prior to 4.13-RC3 I had not been testing any of the new MQ schedulers. I had always only tested with [none]

The tests were with [none] and [mq-deadline]

The new issue I started seeing was not yet reported yet as I was still investigating it.

In summary:
With buffered writes we see the clone fail in blk_get_request in both Mings kernel and in the upstream 4.13-RC3 stock kernel

[  885.271451] io scheduler mq-deadline registered
[  898.455442] device-mapper: multipath: blk_get_request() returned -11 -

requeuing

This is due to

multipath_clone_and_map()

/*
* Map cloned requests (request-based multipath)
*/
static int multipath_clone_and_map(struct dm_target *ti, struct request *rq,
union map_info *map_context, struct request **__clone)
{
..
..
..

        clone = blk_get_request(q, rq->cmd_flags | REQ_NOMERGE, GFP_ATOMIC);
        if (IS_ERR(clone)) {
                /* EBUSY, ENODEV or EWOULDBLOCK: requeue */
                bool queue_dying = blk_queue_dying(q);
                DMERR_LIMIT("blk_get_request() returned %ld%s - requeuing",
                            PTR_ERR(clone), queue_dying ? " (path offline)"
: "");
                if (queue_dying) {
                        atomic_inc(&m->pg_init_in_progress);
                        activate_or_offline_path(pgpath);
                        return DM_MAPIO_REQUEUE;
                }
                return DM_MAPIO_DELAY_REQUEUE;
        }

Still investigating but it leads to a hard lockup

So I still need to see if the hard-lockup happens in the stock kernel with mq-deadline and some other work before coming up with a full summary of the issue.

I also intend to re-run all tests including disconnect and reconnect tests on both mq-deadline and none.

Trace below

[ 1553.167357] NMI watchdog: Watchdog detected hard LOCKUP on cpu 4
[ 1553.167359] Modules linked in: mq_deadline binfmt_misc dm_round_robin

xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 rpcrdma ip6table_mangle ip6table_security ip6table_raw iptable_nat ib_isert nf_conntrack_ipv4 iscsi_target_mod nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ib_iser libiscsi scsi_transport_iscsi iptable_mangle iptable_security iptable_raw ebtable_filter ebtables target_core_mod ip6table_filter ip6_tables iptable_filter ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul

[ 1553.167385]  crc32_pclmul ghash_clmulni_intel pcbc aesni_intel sg joydev

ipmi_si hpilo crypto_simd hpwdt iTCO_wdt cryptd ipmi_devintf glue_helper gpio_ich iTCO_vendor_support shpchp ipmi_msghandler pcspkr acpi_power_meter i7core_edac lpc_ich pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_multipath ip_tables xfs libcrc32c sd_mod amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm mlxfw i2c_core ptp serio_raw hpsa crc32c_intel bnx2 pps_core devlink scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_srpt]

[ 1553.167410] CPU: 4 PID: 11532 Comm: dd Tainted: G          I
4.13.0-rc3lobeming.ming_V4+ #20
[ 1553.167411] Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
[ 1553.167412] task: ffff9d9344b0d800 task.stack: ffffb5cb913b0000
[ 1553.167421] RIP: 0010:__blk_recalc_rq_segments+0xec/0x3d0
[ 1553.167421] RSP: 0018:ffff9d9377a83d08 EFLAGS: 00000046
[ 1553.167422] RAX: 0000000000001000 RBX: 0000000000001000 RCX:
ffff9d91b8e9f500
[ 1553.167423] RDX: fffffcd56af20f00 RSI: ffff9d91b8e9f588 RDI:
ffff9d91b8e9f488
[ 1553.167424] RBP: ffff9d9377a83d88 R08: 000000000003c000 R09:
0000000000000001
[ 1553.167424] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 1553.167425] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000003
[ 1553.167426] FS:  00007f3061002740(0000) GS:ffff9d9377a80000(0000)
knlGS:0000000000000000
[ 1553.167427] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1553.167428] CR2: 00007f305a511000 CR3: 0000000b1bfb2000 CR4:
00000000000006e0
[ 1553.167429] Call Trace:
[ 1553.167432]  <IRQ>
[ 1553.167437]  ? mempool_free+0x2b/0x80
[ 1553.167440]  blk_recalc_rq_segments+0x28/0x40
[ 1553.167442]  blk_update_request+0x249/0x310
[ 1553.167450]  end_clone_bio+0x46/0x70 [dm_mod]
[ 1553.167453]  bio_endio+0xa1/0x120
[ 1553.167454]  blk_update_request+0xb7/0x310
[ 1553.167457]  scsi_end_request+0x38/0x1d0
[ 1553.167458]  scsi_io_completion+0x13c/0x630
[ 1553.167460]  scsi_finish_command+0xd9/0x120
[ 1553.167462]  scsi_softirq_done+0x12a/0x150
[ 1553.167463]  __blk_mq_complete_request_remote+0x13/0x20
[ 1553.167466]  flush_smp_call_function_queue+0x52/0x110
[ 1553.167468]  generic_smp_call_function_single_interrupt+0x13/0x30
[ 1553.167470]  smp_call_function_single_interrupt+0x27/0x40
[ 1553.167471]  call_function_single_interrupt+0x93/0xa0
[ 1553.167473] RIP: 0010:radix_tree_next_chunk+0xcb/0x2e0
[ 1553.167474] RSP: 0018:ffffb5cb913b3a68 EFLAGS: 00000203 ORIG_RAX:
ffffffffffffff04
[ 1553.167475] RAX: 0000000000000001 RBX: 0000000000000010 RCX:
ffff9d9e2c556938
[ 1553.167476] RDX: ffff9d93053c5919 RSI: 0000000000000001 RDI:
ffff9d9e2c556910
[ 1553.167476] RBP: ffffb5cb913b3ab8 R08: ffffb5cb913b3bc0 R09:
0000000ab7d52000
[ 1553.167477] R10: fffffcd56adf5440 R11: 0000000000000000 R12:
0000000000000230
[ 1553.167477] R13: 0000000000000000 R14: 000000000006e0fe R15:

ffff9d9e2c556910

Quoted 125 lines · Laurence Oberman

Quoted 100 lines · Ming Lei

In Red Hat internal storage test wrt. blk-mq scheduler, we found that I/O performance is much bad with mq-deadline, especially about sequential I/O on some multi-queue SCSI devcies(lpfc, qla2xxx, SRP...)

Turns out one big issue causes the performance regression: requests are still dequeued from sw queue/scheduler queue even when ldd's queue is busy, so I/O merge becomes quite difficult to make, then sequential IO degrades a lot.

The 1st five patches improve this situation, and brings back some performance loss.

But looks they are still not enough. It is caused by the shared queue depth among all hw queues. For SCSI devices, .cmd_per_lun defines the max number of pending I/O on one request queue, which is per-request_queue depth. So during dispatch, if one hctx is too busy to move on, all hctxs can't dispatch too because of the per-request_queue depth.

Patch 6 ~ 14 use per-request_queue dispatch list to avoid to dequeue requests from sw/scheduler queue when lld queue is busy.

Patch 15 ~20 improve bio merge via hash table in sw queue, which makes bio merge more efficient than current approch in which only the last 8 requests are checked. Since patch 6~14 converts to the scheduler way of dequeuing one request from sw queue one time for SCSI device, and the times of acquring ctx->lock is increased, and merging bio via hash table decreases holding time of ctx->lock and should eliminate effect from patch 14.

With this changes, SCSI-MQ sequential I/O performance is improved much, for lpfc, it is basically brought back compared with block legacy path[1], especially mq-deadline is improved by > X10 [1] on lpfc and by > 3X on SCSI SRP, For mq-none it is improved by 10% on lpfc, and write is improved by > 10% on SRP too.

Also Bart worried that this patchset may affect SRP, so provide test data on SCSI SRP this time:

- fio(libaio, bs:4k, dio, queue_depth:64, 64 jobs)
- system(16 cores, dual sockets, mem: 96G)

               |v4.13-rc3     |v4.13-rc3     | v4.13-rc3+patches |
               |blk-legacy dd |blk-mq none   | blk-mq none  |
-----------------------------------------------------------|
read     :iops|         587K |         526K |         537K |
randread :iops|         115K |         140K |         139K |
write    :iops|         596K |         519K |         602K |
randwrite:iops|         103K |         122K |         120K |

               |v4.13-rc3     |v4.13-rc3     | v4.13-rc3+patches
               |blk-legacy dd |blk-mq dd     | blk-mq dd    |
------------------------------------------------------------
read     :iops|         587K |         155K |         522K |
randread :iops|         115K |         140K |         141K |
write    :iops|         596K |         135K |         587K |
randwrite:iops|         103K |         120K |         118K |

V2:
- dequeue request from sw queues in round roubin's style as suggested by Bart, and introduces one helper in sbitmap for this purpose
- improve bio merge via hash table from sw queue
- add comments about using DISPATCH_BUSY state in lockless way, simplifying handling on busy state,
- hold ctx->lock when clearing ctx busy bit as suggested by Bart

[1] http://marc.info/?l=linux-block&m=150151989915776&w=2

Ming Lei (20):
blk-mq-sched: fix scheduler bad performance
sbitmap: introduce __sbitmap_for_each_set()
blk-mq: introduce blk_mq_dispatch_rq_from_ctx()
blk-mq-sched: move actual dispatching into one helper
blk-mq-sched: improve dispatching from sw queue
blk-mq-sched: don't dequeue request until all in ->dispatch are
flushed
blk-mq-sched: introduce blk_mq_sched_queue_depth()
blk-mq-sched: use q->queue_depth as hint for q->nr_requests
blk-mq: introduce BLK_MQ_F_SHARED_DEPTH
blk-mq-sched: introduce helpers for query, change busy state
blk-mq: introduce helpers for operating ->dispatch list
blk-mq: introduce pointers to dispatch lock & list
blk-mq: pass 'request_queue *' to several helpers of operating BUSY
blk-mq-sched: improve IO scheduling on SCSI devcie
block: introduce rqhash helpers
block: move actual bio merge code into __elv_merge
block: add check on elevator for supporting bio merge via hashtable
from blk-mq sw queue
block: introduce .last_merge and .hash to blk_mq_ctx
blk-mq-sched: refactor blk_mq_sched_try_merge()
blk-mq: improve bio merge from blk-mq sw queue

Quoted 15 lines

  block/blk-mq-debugfs.c  |  12 ++--
  block/blk-mq-sched.c    | 187 +++++++++++++++++++++++++++++-
------------------
  block/blk-mq-sched.h    |  23 ++++++
  block/blk-mq.c          | 133 +++++++++++++++++++++++++++++++---
  block/blk-mq.h          |  73 +++++++++++++++++++
  block/blk-settings.c    |   2 +
  block/blk.h             |  55 ++++++++++++++
  block/elevator.c        |  93 ++++++++++++++----------
  include/linux/blk-mq.h  |   5 ++
  include/linux/blkdev.h  |   5 ++
  include/linux/sbitmap.h |  54 ++++++++++----
  11 files changed, 504 insertions(+), 138 deletions(-)

Hello

I tested this series using Ming's tests as well as my own set of tests typically run against changes to upstream code in my SRP test-bed. My tests also include very large sequential buffered and un-buffered I/O.

Quoted 8 lines

This series seems to be fine for me. I did uncover another issue that is unrelated to these patches and also exists in 4.13-RC3 generic that I am still debugging.

For what its worth:

Tested-by Laurence Oberman <[email protected]>

Hello

I need to retract my Tested-by:

While its valid that the patches do not introduce performance regressions, they seem to cause a hard lockup when the [mq-deadline] scheduler is enabled so I am not confident with a passing result here.

This is specific to large buffered I/O writes (4MB) At least that is my current test.

I did not wait long enough for the issue to show when I first sent the pass (Tested-by) message because I know my test platform so well I thought I had given it enough time to validate the patches for performance regressions.

I dont know if the failing clone in blk_get_request() is a direct a catalyst for the hard lockup but what I do know is with the stock upstream 4.13-RC3 I only see them when I am set to [none] and stock upstream never seems to see the hard lockup.

With [mq-deadline] enabled on stock I dont see them at all and it behaves.

Now with Ming's patches if we enable [mq-deadline] we DO see the clone failures

and the hard lockup so we have opposit behaviour with the scheduler choice and we have the hard lockup.

On Ming's kernel with [none] we are well behaved and that was my original focus, testing on [none] and hence my Tested-by: pass.

On Mon, Aug 7, 2017 at 8:48 AM, Laurence Oberman <[email protected]> wrote:
Hello

I need to retract my Tested-by:

While its valid that the patches do not introduce performance regressions, they seem to cause a hard lockup when the [mq-deadline] scheduler is enabled so I am not confident with a passing result here.

This is specific to large buffered I/O writes (4MB) At least that is my current test.

I did not wait long enough for the issue to show when I first sent the pass (Tested-by) message because I know my test platform so well I thought I had given it enough time to validate the patches for performance regressions.

I dont know if the failing clone in blk_get_request() is a direct a catalyst for the hard lockup but what I do know is with the stock upstream 4.13-RC3 I only see them when I am set to [none] and stock upstream never seems to see the hard lockup.

With [mq-deadline] enabled on stock I dont see them at all and it behaves.