Skip to contentSkip to conversation

mmc: cqhci: commit descriptors before setting the doorbell

RFC In review Faiz Abbas 3 participants 8 messages Oct 14, 2019 – Oct 18, 2019

Subsystems (2)
lore
Share & tools
lore

Faiz Abbas

author
#0 lore raw

RFC 1/1 mmc: cqhci: commit descriptors before setting the doorbell

Add a write memory barrier to make sure that descriptors are actually written to memory before ringing the doorbell.

---

This patch fixes a very infrequent ADMA error (1 out of 100 times) that I have been seeing after enabling command queuing for J721e. Also looking at memory-barriers.txt and this commit[1],
it looks like we should be doing this before any descriptor write followed by a doorbell ring operation. It'll be nice if someone with more expertise in memory barriers can comment.

[1] ad1a1b9cd67a ("scsi: ufs: commit descriptors before setting the doorbell")

 drivers/mmc/host/cqhci.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/mmc/host/cqhci.c +2 -1
index f7bdae5354c3..5047f7343ffc 100644
@@ -611,7 +611,8 @@ static int cqhci_request(struct mmc_host *mmc, struct mmc_request *mrq)
 	cq_host->slot[tag].flags = 0;
 
 	cq_host->qcnt += 1;
-
+	/* Make sure descriptors are ready before ringing the doorbell */
+	wmb();
 	cqhci_writel(cq_host, 1 << tag, CQHCI_TDBR);
 	if (!(cqhci_readl(cq_host, CQHCI_TDBR) & (1 << tag)))
 		pr_debug("%s: cqhci: doorbell not set for tag %d\n",
Signature · 1 line
2.19.2
author
#1 lore raw

Hi,

Quoted 13 lines · Faiz Abbas

Add a write memory barrier to make sure that descriptors are actually written to memory before ringing the doorbell.

---

This patch fixes a very infrequent ADMA error (1 out of 100 times) that I have been seeing after enabling command queuing for J721e. Also looking at memory-barriers.txt and this commit[1],
it looks like we should be doing this before any descriptor write followed by a doorbell ring operation. It'll be nice if someone with more expertise in memory barriers can comment.

Quoted 2 lines

[1] ad1a1b9cd67a ("scsi: ufs: commit descriptors before setting the doorbell")

So I see that cqhci_readl/writel() use readl/writel_relaxed() which seems to be causing this issue. Should I just fix this by converting those to readl/writel with memory barriers instead?

Thanks,
Faiz

maintainer
#2 lore raw
Quoted 16 lines · Faiz Abbas

Hi,

Quoted 13 lines · Faiz Abbas

Add a write memory barrier to make sure that descriptors are actually written to memory before ringing the doorbell.

---

This patch fixes a very infrequent ADMA error (1 out of 100 times) that I have been seeing after enabling command queuing for J721e. Also looking at memory-barriers.txt and this commit[1],
it looks like we should be doing this before any descriptor write followed by a doorbell ring operation. It'll be nice if someone with more expertise in memory barriers can comment.

Quoted 6 lines
Quoted 2 lines

[1] ad1a1b9cd67a ("scsi: ufs: commit descriptors before setting the doorbell")

So I see that cqhci_readl/writel() use readl/writel_relaxed() which seems to be causing this issue. Should I just fix this by converting those to readl/writel with memory barriers instead?

Perhaps we could do both changes i.e. add wmb() and convert to non-relaxed readl/writel

author
#3 lore raw

Adrian,

Quoted 20 lines · Adrian Hunter
Quoted 19 lines · Faiz Abbas

Hi,

Quoted 15 lines · Faiz Abbas

Add a write memory barrier to make sure that descriptors are actually written to memory before ringing the doorbell.

---

This patch fixes a very infrequent ADMA error (1 out of 100 times) that I have been seeing after enabling command queuing for J721e. Also looking at memory-barriers.txt and this commit[1],
it looks like we should be doing this before any descriptor write followed by a doorbell ring operation. It'll be nice if someone with more expertise in memory barriers can comment.

[1] ad1a1b9cd67a ("scsi: ufs: commit descriptors before setting the doorbell")

Quoted 7 lines
Quoted 3 lines

So I see that cqhci_readl/writel() use readl/writel_relaxed() which seems to be causing this issue. Should I just fix this by converting those to readl/writel with memory barriers instead?

Perhaps we could do both changes i.e. add wmb() and convert to non-relaxed readl/writel

readl is implemented as readl_relaxed(); __rmb(); and
writel is implemented as wmb(); writel_relaxed();

I think another wmb() before writel will be redundant.

Maybe this patch is good enough in itself.

Thanks,
Faiz

author
#4 lore raw

Adrian,

Quoted 31 lines · Faiz Abbas

Adrian,

Quoted 20 lines · Adrian Hunter
Quoted 19 lines · Faiz Abbas

Hi,

Quoted 15 lines · Faiz Abbas

Add a write memory barrier to make sure that descriptors are actually written to memory before ringing the doorbell.

---

This patch fixes a very infrequent ADMA error (1 out of 100 times) that I have been seeing after enabling command queuing for J721e. Also looking at memory-barriers.txt and this commit[1],
it looks like we should be doing this before any descriptor write followed by a doorbell ring operation. It'll be nice if someone with more expertise in memory barriers can comment.

[1] ad1a1b9cd67a ("scsi: ufs: commit descriptors before setting the doorbell")

Quoted 7 lines
Quoted 3 lines

So I see that cqhci_readl/writel() use readl/writel_relaxed() which seems to be causing this issue. Should I just fix this by converting those to readl/writel with memory barriers instead?

Perhaps we could do both changes i.e. add wmb() and convert to non-relaxed readl/writel

Quoted 8 lines

readl is implemented as readl_relaxed(); __rmb(); and
writel is implemented as wmb(); writel_relaxed();

I think another wmb() before writel will be redundant.

Maybe this patch is good enough in itself.

Do you agree?

Thanks,
Faiz

maintainer Acked-by
#5 lore raw
Quoted 38 lines · Faiz Abbas

Adrian,

Quoted 35 lines · Faiz Abbas

Adrian,

Quoted 20 lines · Adrian Hunter
Quoted 19 lines · Faiz Abbas

Hi,

Quoted 15 lines · Faiz Abbas

Add a write memory barrier to make sure that descriptors are actually written to memory before ringing the doorbell.

---

This patch fixes a very infrequent ADMA error (1 out of 100 times) that I have been seeing after enabling command queuing for J721e. Also looking at memory-barriers.txt and this commit[1],
it looks like we should be doing this before any descriptor write followed by a doorbell ring operation. It'll be nice if someone with more expertise in memory barriers can comment.

[1] ad1a1b9cd67a ("scsi: ufs: commit descriptors before setting the doorbell")

Quoted 7 lines
Quoted 3 lines

So I see that cqhci_readl/writel() use readl/writel_relaxed() which seems to be causing this issue. Should I just fix this by converting those to readl/writel with memory barriers instead?

Perhaps we could do both changes i.e. add wmb() and convert to non-relaxed readl/writel

readl is implemented as readl_relaxed(); __rmb(); and
writel is implemented as wmb(); writel_relaxed();

Quoted 6 lines
Quoted 4 lines

I think another wmb() before writel will be redundant.

Maybe this patch is good enough in itself.

Do you agree?

Sure.

#6 lore raw
In reply to Faiz Abbas
Quoted 5 lines · Faiz Abbas

Add a write memory barrier to make sure that descriptors are actually written to memory before ringing the doorbell.

Applied for fixes and by adding a stable tag, thanks!

BTW, do you have a valid commit that it fixes?

Kind regards
Uffe

Quoted 32 lines

---

This patch fixes a very infrequent ADMA error (1 out of 100 times) that I have been seeing after enabling command queuing for J721e. Also looking at memory-barriers.txt and this commit[1],
it looks like we should be doing this before any descriptor write followed by a doorbell ring operation. It'll be nice if someone with more expertise in memory barriers can comment.

[1] ad1a1b9cd67a ("scsi: ufs: commit descriptors before setting the doorbell")

 drivers/mmc/host/cqhci.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/mmc/host/cqhci.c +2 -2
index f7bdae5354c3..5047f7343ffc 100644
@@ -611,7 +611,8 @@ static int cqhci_request(struct mmc_host *mmc, struct mmc_request *mrq)
        cq_host->slot[tag].flags = 0;
 
        cq_host->qcnt += 1;
-
+       /* Make sure descriptors are ready before ringing the doorbell */
+       wmb();
        cqhci_writel(cq_host, 1 << tag, CQHCI_TDBR);
        if (!(cqhci_readl(cq_host, CQHCI_TDBR) & (1 << tag)))
                pr_debug("%s: cqhci: doorbell not set for tag %d\n",
--

2.19.2

author
#7 lore raw

Hi Uffe,

Quoted 11 lines · Ulf Hansson
Quoted 5 lines · Faiz Abbas

Add a write memory barrier to make sure that descriptors are actually written to memory before ringing the doorbell.

Applied for fixes and by adding a stable tag, thanks!

BTW, do you have a valid commit that it fixes?

You can add:

Fixes a4080225f51d ("mmc: cqhci: support for command queue enabled host")

Thanks,
Faiz