Skip to contentSkip to conversation

nvmet-rdma: fix null dereference under heavy load

PATCH Superseded Raju Rangoju 5 participants 8 messages Jan 3, 2019 – Jan 21, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 nvmet-rdma: fix null dereference under heavy load

Under heavy load if we don't have any pre-allocated rsps left, we dynamically allocate a rsp, but we are not actually allocating memory for nvme_completion (rsp->req.rsp). In such a case, accessing pointer fields (req->rsp->status) in nvmet_req_init() will result in crash.

To fix this, allocate the memory for nvme_completion by calling nvmet_rdma_alloc_rsp()

Fixes 8407879c("nvmet-rdma:fix possible bogus dereference under heavy load")
Cc <stable at vger.kernel.org>
Reviewed-by Max Gurtovoy <maxg at mellanox.com>
Signed-off-by Raju Rangoju <rajur at chelsio.com>

--
Changes from v2:
- Dropped the unneeded initialization of ret variable

Changes from v1:
- Moved integer to 'if' block
- Used 'unlikely' in datapath flow condition

---
 drivers/nvme/target/rdma.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +14 -1
index a8d23eb80192..a884e3a0e8af 100644
@@ -139,6 +139,10 @@ static void nvmet_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc);
 static void nvmet_rdma_read_data_done(struct ib_cq *cq, struct ib_wc *wc);
 static void nvmet_rdma_qp_event(struct ib_event *event, void *priv);
 static void nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue);
+static void nvmet_rdma_free_rsp(struct nvmet_rdma_device *ndev,
+				struct nvmet_rdma_rsp *r);
+static int nvmet_rdma_alloc_rsp(struct nvmet_rdma_device *ndev,
+				struct nvmet_rdma_rsp *r);
 
 static const struct nvmet_fabrics_ops nvmet_rdma_ops;
 
@@ -182,9 +186,17 @@ nvmet_rdma_get_rsp(struct nvmet_rdma_queue *queue)
 	spin_unlock_irqrestore(&queue->rsps_lock, flags);
 
 	if (unlikely(!rsp)) {
-		rsp = kmalloc(sizeof(*rsp), GFP_KERNEL);
+		int ret;
+
+		rsp = kzalloc(sizeof(*rsp), GFP_KERNEL);
 		if (unlikely(!rsp))
 			return NULL;
+		ret = nvmet_rdma_alloc_rsp(queue->dev, rsp);
+		if (unlikely(ret)) {
+			kfree(rsp);
+			return NULL;
+		}
+
 		rsp->allocated = true;
 	}
 
@@ -197,6 +209,7 @@ nvmet_rdma_put_rsp(struct nvmet_rdma_rsp *rsp)
 	unsigned long flags;
 
 	if (unlikely(rsp->allocated)) {
+		nvmet_rdma_free_rsp(rsp->queue->dev, rsp);
 		kfree(rsp);
 		return;
 	}
Signature · 1 line
2.12.0
maintainer
#1 lore raw

PATCH v3 1/1 nvmet-rdma: fix null dereference under heavy load

Wonder how the original fix got a tested-by...

Reviewed-by Sagi Grimberg <sagi at grimberg.me>
#2 lore raw

PATCH v3 1/1 nvmet-rdma: fix null dereference under heavy load

Quoted 3 lines · Sagi Grimberg

Wonder how the original fix got a tested-by...

That was my bad.  But the testers said they couldn't reproduce the issue, so I assumed that meant it was fixed.  I guess they weren't stressing it enough.

Steve.

#3 lore raw

PATCH v3 1/1 nvmet-rdma: fix null dereference under heavy load

Hi Christoph,

In case you missed, could you please pull this patch ?

-Raju

-----Original Message-----
From: Steve Wise <[email protected]>
Sent: 07 January 2019 21:29
To: Sagi Grimberg <sagi at grimberg.me>; Raju Rangoju <rajur at chelsio.com>; linux-nvme at lists.infradead.org
Cc: maxg at mellanox.com; stable at vger.kernel.org
Subject: Re: [PATCH v3] nvmet-rdma: fix null dereference under heavy load

Quoted 3 lines · Sagi Grimberg

Wonder how the original fix got a tested-by...

Reviewed-by Sagi Grimberg <sagi at grimberg.me>

That was my bad.? But the testers said they couldn't reproduce the issue, so I assumed that meant it was fixed.? I guess they weren't stressing it enough.

Steve.

#4 lore raw

PATCH v3 1/1 nvmet-rdma: fix null dereference under heavy load

In reply to Raju Rangoju

This looks ok to me:

Reviewed-by Christoph Hellwig <hch at lst.de>

In the long run I wonder if we should get rid of the ?tatic preallocation entirely and just use a free pool to avoid constant reallocations..

maintainer Applied
#5 lore raw

PATCH v3 1/1 nvmet-rdma: fix null dereference under heavy load

Quoted 3 lines

This looks ok to me:

Reviewed-by Christoph Hellwig <hch at lst.de>

Applied to nvme-5.0

Quoted 3 lines

In the long run I wonder if we should get rid of the ?tatic preallocation entirely and just use a free pool to avoid constant reallocations..

you mean a kmem_cache? very possible, we can also make the command struct to contain the rsp behind it to avoid the extra allocation...

#6 lore raw

PATCH v3 1/1 nvmet-rdma: fix null dereference under heavy load

Quoted 4 lines · Sagi Grimberg
Quoted 3 lines

This looks ok to me:

Reviewed-by Christoph Hellwig <hch at lst.de>
Quoted 8 lines

Applied to nvme-5.0

Quoted 3 lines

In the long run I wonder if we should get rid of the ?tatic preallocation entirely and just use a free pool to avoid constant reallocations..

you mean a kmem_cache? very possible, we can also make the command struct to contain the rsp behind it to avoid the extra allocation...

yes, the container idea sounds good..

#7 lore raw

PATCH v3 1/1 nvmet-rdma: fix null dereference under heavy load

In reply to Sagi Grimberg
Quoted 4 lines · Sagi Grimberg
Quoted 3 lines

This looks ok to me:

Reviewed-by Christoph Hellwig <hch at lst.de>
Quoted 8 lines

Applied to nvme-5.0

Quoted 3 lines

In the long run I wonder if we should get rid of the ?tatic preallocation entirely and just use a free pool to avoid constant reallocations..

you mean a kmem_cache? very possible, we can also make the command struct to contain the rsp behind it to avoid the extra allocation...

Hey guys,

If every command struct requires a rsp, then it seems simpler to contain it in the command struct.

My 2 cents...

Steve.