Skip to contentSkip to conversation

nvme-pci: dma read memory barrier for completions

PATCH Applied Keith Busch 5 participants 7 messages May 8, 2020 – May 12, 2020 NVMe

lore
Share & tools
lore

Keith Busch

author maintainer
#0 lore raw

PATCH 1/1 nvme-pci: dma read memory barrier for completions

Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.

Commit 324b494c2862 inlined verifying a new completion with its handling. At least one architecture was observed to access the contents out of order, resulting in the driver using stale data for the completion.

Add a dma read barrier before reading the completion queue entry and after the condition its contents depend on to ensure the read order is determinsitic.

---
 drivers/nvme/host/pci.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/nvme/host/pci.c +5 -0
index e13c370de830..3726dc780d15 100644
@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)
 
 	while (nvme_cqe_pending(nvmeq)) {
 		found++;
+		/*
+		 * load-load control dependency between phase and the rest of
+		 * the cqe requires a full read memory barrier
+		 */
+		dma_rmb();
 		nvme_handle_cqe(nvmeq, nvmeq->cq_head);
 		nvme_update_cq_head(nvmeq);
 	}
Signature · 7 lines
2.24.1


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 5 lines

Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.

Commit 324b494c2862 inlined verifying a new completion with its

I think we need a "Fixes" tag to get it into stable 5.6.y right?

Otherwise,

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#2 lore raw
Quoted 7 lines · Sagi Grimberg
Quoted 5 lines

Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.

Commit 324b494c2862 inlined verifying a new completion with its

I think we need a "Fixes" tag to get it into stable 5.6.y right?

5.6 doesn't include the commit that reintroduced inline completion handling, so there's no stable this would apply to.

But really, the driver handled completions inline with the phase for most of its existence (3.3 - 4.17). It was only recently we had the split handling, so I'm not sure why its only an issue now unless more recent compilers and/or CPU made it more likely.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#3 lore raw
Quoted 6 lines · Sagi Grimberg
Quoted 4 lines
Quoted 4 lines

Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Commit 324b494c2862 inlined verifying a new completion with its

I think we need a "Fixes" tag to get it into stable 5.6.y right?

5.6 doesn't include the commit that reintroduced inline completion handling, so there's no stable this would apply to.

From some reason I thought I saw it in 5.6, but you're right it isn't, so you can ignore...

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

Tested-by on v1 1/1
#4 lore raw
In reply to Keith Busch
Quoted 11 lines · Keith Busch

Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.

Commit 324b494c2862 inlined verifying a new completion with its handling. At least one architecture was observed to access the contents out of order, resulting in the driver using stale data for the completion.

Add a dma read barrier before reading the completion queue entry and after the condition its contents depend on to ensure the read order is determinsitic.

deterministic

Quoted 3 lines
Quoted 12 lines
---
  drivers/nvme/host/pci.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/nvme/host/pci.c +0 -0
index e13c370de830..3726dc780d15 100644
@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)
 

drivers/nvme/host/pci.c static inline int nvme_process_cq(struct nvme_queue *nvmeq) · line 999

Quoted 10 lines
drivers/nvme/host/pci.c +5 -0
  	while (nvme_cqe_pending(nvmeq)) {
  		found++;
+		/*
+		 * load-load control dependency between phase and the rest of
+		 * the cqe requires a full read memory barrier
+		 */
+		dma_rmb();
  		nvme_handle_cqe(nvmeq, nvmeq->cq_head);
  		nvme_update_cq_head(nvmeq);
  	}

JFYI, I have confirmed that the ES3000 NVMe controllers which I am using will write all 4 dwords of the CQE at the same time.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

Acked-by on v1 1/1
#5 lore raw
In reply to Keith Busch
Quoted 25 lines · Keith Busch

Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.

Commit 324b494c2862 inlined verifying a new completion with its handling. At least one architecture was observed to access the contents out of order, resulting in the driver using stale data for the completion.

Add a dma read barrier before reading the completion queue entry and after the condition its contents depend on to ensure the read order is determinsitic.

---
 drivers/nvme/host/pci.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/nvme/host/pci.c +0 -0
index e13c370de830..3726dc780d15 100644
@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)
 

drivers/nvme/host/pci.c static inline int nvme_process_cq(struct nvme_queue *nvmeq) · line 999

Quoted 10 lines
drivers/nvme/host/pci.c +5 -0
 	while (nvme_cqe_pending(nvmeq)) {
 		found++;
+		/*
+		 * load-load control dependency between phase and the rest of
+		 * the cqe requires a full read memory barrier
+		 */
+		dma_rmb();
 		nvme_handle_cqe(nvmeq, nvmeq->cq_head);
 		nvme_update_cq_head(nvmeq);
 	}

Thanks for doing this:

Will

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme