nvme-pci: dma read memory barrier for completions
Keith Busch
maintainerPATCH 1/1 nvme-pci: dma read memory barrier for completions
Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.
Commit 324b494c2862 inlined verifying a new completion with its handling. At least one architecture was observed to access the contents out of order, resulting in the driver using stale data for the completion.
Add a dma read barrier before reading the completion queue entry and after the condition its contents depend on to ensure the read order is determinsitic.
--- drivers/nvme/host/pci.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/nvme/host/pci.c +5 -0
@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)while (nvme_cqe_pending (nvmeq)) { found++; +/* +* load-load control dependency between phase and the rest of +* the cqe requires a full read memory barrier +*/ +dma_rmb ();nvme_handle_cqe (nvmeq, nvmeq->cq_head);nvme_update_cq_head (nvmeq); }
Signature · 7 lines
2.24.1 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 5 lines
Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.
Commit
324b494c2862inlined verifying a new completion with its
I think we need a "Fixes" tag to get it into stable 5.6.y right?
Otherwise,
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 7 lines · Sagi Grimberg
Quoted 5 lines
Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.
Commit
324b494c2862inlined verifying a new completion with itsI think we need a "Fixes" tag to get it into stable 5.6.y right?
5.6 doesn't include the commit that reintroduced inline completion handling, so there's no stable this would apply to.
But really, the driver handled completions inline with the phase for most of its existence (3.3 - 4.17). It was only recently we had the split handling, so I'm not sure why its only an issue now unless more recent compilers and/or CPU made it more likely.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 6 lines · Sagi Grimberg
Quoted 4 lines
Quoted 4 lines
Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Commit
324b494c2862inlined verifying a new completion with itsI think we need a "Fixes" tag to get it into stable 5.6.y right?
5.6 doesn't include the commit that reintroduced inline completion handling, so there's no stable this would apply to.
From some reason I thought I saw it in 5.6, but you're right it isn't, so you can ignore...
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 11 lines · Keith Busch
Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.
Commit
324b494c2862inlined verifying a new completion with its handling. At least one architecture was observed to access the contents out of order, resulting in the driver using stale data for the completion.Add a dma read barrier before reading the completion queue entry and after the condition its contents depend on to ensure the read order is determinsitic.
deterministic
Quoted 3 lines
Reported-by John Garry <[email protected]>Suggested-by Will Deacon <[email protected]>
Quoted 12 lines
Signed-off-by Keith Busch <[email protected]>--- drivers/nvme/host/pci.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/nvme/host/pci.c +0
-0@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)
drivers/nvme/host/pci.c static inline int nvme_process_cq(struct nvme_queue *nvmeq) · line 999
Quoted 10 lines
drivers/nvme/host/pci.c +5
-0while (nvme_cqe_pending (nvmeq)) { found++; +/* +* load-load control dependency between phase and the rest of +* the cqe requires a full read memory barrier +*/ +dma_rmb ();nvme_handle_cqe (nvmeq, nvmeq->cq_head);nvme_update_cq_head (nvmeq); }
JFYI, I have confirmed that the ES3000 NVMe controllers which I am using will write all 4 dwords of the CQE at the same time.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 25 lines · Keith Busch
Control dependencies do not guarantee load order across the condition, allowing a CPU to predict and speculate memory reads.
Commit
324b494c2862inlined verifying a new completion with its handling. At least one architecture was observed to access the contents out of order, resulting in the driver using stale data for the completion.Add a dma read barrier before reading the completion queue entry and after the condition its contents depend on to ensure the read order is determinsitic.
Reported-by John Garry <[email protected]>Suggested-by Will Deacon <[email protected]>Signed-off-by Keith Busch <[email protected]>--- drivers/nvme/host/pci.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/nvme/host/pci.c +0
-0@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)
drivers/nvme/host/pci.c static inline int nvme_process_cq(struct nvme_queue *nvmeq) · line 999
Quoted 10 lines
drivers/nvme/host/pci.c +5
-0while (nvme_cqe_pending (nvmeq)) { found++; +/* +* load-load control dependency between phase and the rest of +* the cqe requires a full read memory barrier +*/ +dma_rmb ();nvme_handle_cqe (nvmeq, nvmeq->cq_head);nvme_update_cq_head (nvmeq); }
Thanks for doing this:
Will
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Thanks,
applied to nvme-5.7.
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme