Patch "nvme-rdma: support devices with queue size < 32" has been added to the 4.11-stable tree
Subsystems (2)
This is a note to let you know that I've just added the patch titled
nvme-rdma: support devices with queue size < 32
to the 4.11-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
nvme-rdma-support-devices-with-queue-size-32.patch
and it can be found in the queue-4.11 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.
Quoted 1 line
From
0544f5494a03b8846db74e02be5685d1f32b06c9Mon Sep 17 00:00:00 2001
From: Marta Rybczynska <[email protected]>
Date: Mon, 10 Apr 2017 17:12:34 +0200
Subject: nvme-rdma: support devices with queue size < 32
From: Marta Rybczynska <[email protected]>
commit 0544f5494a03b8846db74e02be5685d1f32b06c9 upstream.
In the case of small NVMe-oF queue size (<32) we may enter a deadlock caused by the fact that the IB completions aren't sent waiting for 32 and the send queue will fill up.
The error is seen as (using mlx5):
[ 2048.693355] mlx5_0:mlx5_ib_post_send:3765:(pid 7273): [ 2048.693360] nvme nvme1: nvme_rdma_post_send failed with error code -12
This patch changes the way the signaling is done so that it depends on the queue depth now. The magic define has been removed completely.
--- drivers/nvme/host/rdma.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/nvme/host/rdma.c +14 -4
@@ -1029,6 +1029,19 @@ static void nvme_rdma_send_done(struct invme_rdma_wr_error (cq, wc,"SEND" ); } +static inline int nvme_rdma_queue_sig_limit (struct nvme_rdma_queue *queue) +{ +int sig_limit; + +/* +* We signal completion every queue depth/2 and also handle the +* degenerated case of a device with queue_depth=1, where we +* would need to signal every message. +*/ + sig_limit =max (queue->queue_size /2 ,1 ); +return (++queue->sig_count % sig_limit) ==0 ; +} +static int nvme_rdma_post_send (struct nvme_rdma_queue *queue,struct nvme_rdma_qe *qe,struct ib_sge *sge,u32 num_sge,struct ib_send_wr *first,bool flush)
@@ -1056,9 +1069,6 @@ static int nvme_rdma_post_send(struct nv* Would have been way to obvious to handle this in hardware or * at least the RDMA stack.. * -* This messy and racy code sniplet is copy and pasted from the iSER -* initiator, and the magic '32' comes from there as well. -* * Always signal the flushes. The magic request used for the flush * sequencer is not allocated in our driver's tagset and it's * triggered to be freed by blk_cleanup_queue(). So we need to
@@ -1066,7 +1076,7 @@ static int nvme_rdma_post_send(struct nv* embedded in request's payload, is not freed when __ib_process_cq() * calls wr_cqe->done(). */ -if ((++queue->sig_count %32 ) ==0 || flush) +if (nvme_rdma_queue_sig_limit (queue) || flush) wr.send_flags |=IB_SEND_SIGNALED ;if (first)
Patches currently in stable-queue which might be from [email protected] are
queue-4.11/nvme-rdma-support-devices-with-queue-size-32.patch