Skip to contentSkip to conversation

stmmac barrier fixes and cleanup

PATCH In review Niklas Cassel 3 participants 14 messages Feb 26, 2018 – Mar 8, 2018

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover stmmac barrier fixes and cleanup

stmmac barrier fixes and cleanup

Niklas Cassel (4):
net: stmmac: ensure that the MSS desc is the last desc to set the own
bit
net: stmmac: use correct barrier between coherent memory and MMIO
net: stmmac: ensure that the device has released ownership before
reading data
net: stmmac: make dwmac4_release_tx_desc() clear all descriptor fields

 drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c |  2 ++
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c  | 18 +++++++++++++++---
 2 files changed, 17 insertions(+), 3 deletions(-)
Signature · 1 line
2.14.2
#1 lore raw

PATCH 1/4 net: stmmac: ensure that the MSS desc is the last desc to set the own bit

A dma_wmb() is used to guarantee the ordering, with respect to other writes, to cache coherent DMA memory.

There is a dma_wmb() in prepare_tx_desc()/prepare_tso_tx_desc() which ensures that TDES0/1/2 is written before TDES3 (which contains the own bit), for First Desc.

However, in the rare case that MSS changes, there will be a MSS context descriptor in front of the regular DMA descriptors:

<MSS desc> <- DMA Next Descriptor
<First Desc>
<desc n>
<Last Desc>

Thus, for this special case, we need a dma_wmb()
after prepare_tso_tx_desc()/before writing the own bit to the MSS desc, so that we flush the write to TDES3 for First Desc,
in order to ensure that the MSS descriptor is the last descriptor to set the own bit.

---
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +8 -1
index c8d86d77e03d..3b5e7b06e796 100644
@@ -2983,8 +2983,15 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
 			tcp_hdrlen(skb) / 4, (skb->len - proto_hdr_len));
 
 	/* If context desc is used to change MSS */
-	if (mss_desc)
+	if (mss_desc) {
+		/* Make sure that first descriptor has been completely
+		 * written, including its own bit. This is because MSS is
+		 * actually before first descriptor, so we need to make
+		 * sure that MSS's own bit is the last thing written.
+		 */
+		dma_wmb();
 		priv->hw->desc->set_tx_owner(mss_desc);
+	}
 
 	/* The own bit must be the latest setting done when prepare the
 	 * descriptor and then barrier is needed to make sure that
Signature · 1 line
2.14.2
#2 lore raw

PATCH 2/4 net: stmmac: use correct barrier between coherent memory and MMIO

In reply to Niklas Cassel

The last memory barrier in stmmac_xmit()/stmmac_tso_xmit() is placed between a coherent memory write and a MMIO write:

The own bit is written in First Desc (TSO: MSS desc or First Desc). <barrier>
The DMA engine is started by a write to the tx desc tail pointer/ enable dma transmission register, i.e. a MMIO write.

This barrier cannot be a simple dma_wmb(), since a dma_wmb() is only used to guarantee the ordering, with respect to other writes, to cache coherent DMA memory.

To guarantee that the cache coherent memory writes have completed before we attempt to write to the cache incoherent MMIO region, we need to use the more heavyweight barrier wmb().

---
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +2 -2
index 3b5e7b06e796..6dd04f237b2a 100644
@@ -2997,7 +2997,7 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
 	 * descriptor and then barrier is needed to make sure that
 	 * all is coherent before granting the DMA engine.
 	 */
-	dma_wmb();
+	wmb();
 
 	if (netif_msg_pktdata(priv)) {
 		pr_info("%s: curr=%d dirty=%d f=%d, e=%d, f_p=%p, nfrags %d\n",
@@ -3221,7 +3221,7 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
 		 * descriptor and then barrier is needed to make sure that
 		 * all is coherent before granting the DMA engine.
 		 */
-		dma_wmb();
+		wmb();
 	}
 
 	netdev_tx_sent_queue(netdev_get_tx_queue(dev, queue), skb->len);
Signature · 1 line
2.14.2
#3 lore raw

Re: [PATCH net-next 2/4] net: stmmac: use correct barrier between coherent memory and MMIO

Hi!

Thanks for doing the detective work!

Quoted 3 lines

This barrier cannot be a simple dma_wmb(), since a dma_wmb() is only used to guarantee the ordering, with respect to other writes, to cache coherent DMA memory.

Could you explain this a bit more (and perhaps in code comment)?

Ensuring other writes are done before writing the "GO!" bit should be enough, no?

(If it is not, do we need heavier barriers in other places, too?)

Best regards,
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
maintainer
#4 lore raw

Re: [PATCH net-next 2/4] net: stmmac: use correct barrier between coherent memory and MMIO

From: Pavel Machek <[email protected]>
Date: Fri, 2 Mar 2018 10:20:00 +0100

Quoted 8 lines
Quoted 3 lines

This barrier cannot be a simple dma_wmb(), since a dma_wmb() is only used to guarantee the ordering, with respect to other writes, to cache coherent DMA memory.

Could you explain this a bit more (and perhaps in code comment)?

Ensuring other writes are done before writing the "GO!" bit should be enough, no?

Indeed, the chip should never look at the descriptor contents unless the GO bit is set.

If there are ways that it can, this must be explained and documented since it is quite unusual compared to other hardware.

Quoted 1 line

(If it is not, do we need heavier barriers in other places, too?)

Right.

#5 lore raw

Re: [PATCH net-next 2/4] net: stmmac: use correct barrier between coherent memory and MMIO

Quoted 3 lines · David Miller

From: Pavel Machek <[email protected]>
Date: Fri, 2 Mar 2018 10:20:00 +0100

Hello Pavel, David

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

This barrier cannot be a simple dma_wmb(), since a dma_wmb() is only used to guarantee the ordering, with respect to other writes, to cache coherent DMA memory.

Could you explain this a bit more (and perhaps in code comment)?

Have a look at:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/memory-barriers.txt?h=v4.16-rc1#n1913

AFAICT, a dma_wmb() can only be used to guarantee that the writes to cache coherent memory (e.g. memory allocated with dma_alloc_coherent()) before the dma_wmb() will be performed before the writes to cache coherent memory after the dma_wmb().

Since most of our writes are simply writing new buffer addresses and sizes to TDES0/TDES1/TDES2/TDES3, and since these TX DMA descriptors have been allocated with dma_alloc_coherent(), a dma_wmb() should be enough to e.g. make sure that TDES3 (which contains the OWN bit), is written after the writes to TDES0/TDES1/TDES2.

However, the last write we do is "DMA start transmission", this is a register in the IP, i.e. it is a write to the cache incoherent MMIO region (rather than a write to cache coherent memory). To ensure that all writes to cache coherent memory have completed before we start the DMA, we have to use the barrier wmb() (which performs a more extensive flush compared to dma_wmb()).

So the only place where we have to use a wmb() instead of a dma_wmb() is where we have a write to coherent memory, followed by a write to cache incoherent MMIO.
The only obvious place where we have this situtation is where we write the OWN bit immediately followed by a write to the "DMA start transmission" register.

Note that this also matches how it's done in other other drivers:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/amd/xgbe/xgbe-dev.c?h=v4.16-rc1#n1638

There is already a comment describing the barrier in stmmac_xmit() and stmmac_tso_xmit() that says:
/* The own bit must be the latest setting done when prepare the
* descriptor and then barrier is needed to make sure that
* all is coherent before granting the DMA engine.
*/
However, if you want, we could mention wmb() explicitly in this comment.

Quoted 7 lines
Quoted 3 lines

Ensuring other writes are done before writing the "GO!" bit should be enough, no?

Indeed, the chip should never look at the descriptor contents unless the GO bit is set.

Quoted 6 lines

If there are ways that it can, this must be explained and documented since it is quite unusual compared to other hardware.

Quoted 1 line

(If it is not, do we need heavier barriers in other places, too?)

Right.

I hope that my explaination above has cleared any potential confusion.

Best regards,
Niklas

maintainer
#6 lore raw

Re: [PATCH net-next 2/4] net: stmmac: use correct barrier between coherent memory and MMIO

From: Niklas Cassel <[email protected]>
Date: Sat, 3 Mar 2018 00:28:53 +0100

Quoted 7 lines

However, the last write we do is "DMA start transmission", this is a register in the IP, i.e. it is a write to the cache incoherent MMIO region (rather than a write to cache coherent memory). To ensure that all writes to cache coherent memory have completed before we start the DMA, we have to use the barrier wmb() (which performs a more extensive flush compared to dma_wmb()).

The is an implicit memory barrier between physical memory writes and those to MMIO register space.

So as long as you place the dma_wmb() to ensure the correct ordering within the descriptor words, you need nothing else after the last descriptor word write.

#7 lore raw

Re: [PATCH net-next 2/4] net: stmmac: use correct barrier between coherent memory and MMIO

Quoted 11 lines · David Miller

From: Niklas Cassel <[email protected]>
Date: Sat, 3 Mar 2018 00:28:53 +0100

Quoted 7 lines

However, the last write we do is "DMA start transmission", this is a register in the IP, i.e. it is a write to the cache incoherent MMIO region (rather than a write to cache coherent memory). To ensure that all writes to cache coherent memory have completed before we start the DMA, we have to use the barrier wmb() (which performs a more extensive flush compared to dma_wmb()).

Quoted 6 lines

The is an implicit memory barrier between physical memory writes and those to MMIO register space.

So as long as you place the dma_wmb() to ensure the correct ordering within the descriptor words, you need nothing else after the last descriptor word write.

Hello David,

Looking at writel() in e.g. arch/arm/include/asm/io.h:
#define writel(v,c) ({ __iowmb(); writel_relaxed(v,c); })
it indeed has a __iowmb() (which is defined as a wmb()) in its definition.

Is is safe to assume that this is true for all archs?

If so, perhaps the example at:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/memory-barriers.txt?h=v4.16-rc1#n1913
Should be updated.

Considering this, you can drop/revert:
95eb930a40a0 ("net: stmmac: use correct barrier between coherent memory and MMIO") or perhaps you want me to send a revert?

After reverting 95eb930a40a0, we will still have a dma_wmb() _after_ the last descriptor word write. You just explained that nothing else is needed after the last descriptor word write, so I actually think that this last barrier is superfluous.

Best regards,
Niklas

maintainer
#8 lore raw

Re: [PATCH net-next 2/4] net: stmmac: use correct barrier between coherent memory and MMIO

From: Niklas Cassel <[email protected]>
Date: Wed, 7 Mar 2018 18:21:57 +0100

Quoted 3 lines

Considering this, you can drop/revert:
95eb930a40a0 ("net: stmmac: use correct barrier between coherent memory and MMIO") or perhaps you want me to send a revert?

You must submit explicit patches to do a revert or any other change.

Quoted 4 lines

After reverting 95eb930a40a0, we will still have a dma_wmb() _after_ the last descriptor word write. You just explained that nothing else is needed after the last descriptor word write, so I actually think that this last barrier is superfluous.

You don't need one after the last descriptor write.

Look, you're only concerned with ordering within the descriptor writes.

So it's only about:

	desc->a = x;

	/* Write to 'a' must be visible to the hardware before 'b'. */
	dma_wmb();
	desc->b = y;

	writel();

That's all that you need.

#9 lore raw

Re: [PATCH net-next 2/4] net: stmmac: use correct barrier between coherent memory and MMIO

Quoted 28 lines · David Miller

From: Niklas Cassel <[email protected]>
Date: Wed, 7 Mar 2018 18:21:57 +0100

Quoted 3 lines

Considering this, you can drop/revert:
95eb930a40a0 ("net: stmmac: use correct barrier between coherent memory and MMIO") or perhaps you want me to send a revert?

You must submit explicit patches to do a revert or any other change.

Quoted 4 lines

After reverting 95eb930a40a0, we will still have a dma_wmb() _after_ the last descriptor word write. You just explained that nothing else is needed after the last descriptor word write, so I actually think that this last barrier is superfluous.

You don't need one after the last descriptor write.

Look, you're only concerned with ordering within the descriptor writes.

So it's only about:

	desc->a = x;

	/* Write to 'a' must be visible to the hardware before 'b'. */
	dma_wmb();
	desc->b = y;

	writel();
Quoted 1 line

That's all that you need.

It seems like the first commit that added a wmb()
after set_tx_owner() on first desc (which is be the absolute last mem write) was the following commit:

commit 8e83989106562326bfd6aaf92174fe138efd026b

Author: Deepak Sikri <[email protected]>
Date:   Sun Jul 8 21:14:45 2012 +0000

stmmac: Fix for nfs hang on multiple reboot

It was observed that during multiple reboots nfs hangs. The status of receive descriptors shows that all the descriptors were in control of CPU, and none were assigned to DMA.
Also the DMA status register confirmed that the Rx buffer is unavailable.

This patch adds the fix for the same by adding the memory barriers to ascertain that the all instructions before enabling the Rx or Tx DMA are completed which involves the proper setting of the ownership bit in DMA descriptors.

drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +3 -0
index 51b3b68528ee..ea3003edde18 100644
@@ -1212,6 +1212,7 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
                priv->hw->desc->prepare_tx_desc(desc, 0, len, csum_insertion);
                wmb();
                priv->hw->desc->set_tx_owner(desc);
+               wmb();
        }
 
        /* Interrupt on completition only for the latest segment */
@@ -1227,6 +1228,7 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
 
        /* To avoid raise condition */
        priv->hw->desc->set_tx_owner(first);
+       wmb();
 
        priv->cur_tx++;
 
@@ -1290,6 +1292,7 @@ static inline void stmmac_rx_refill(struct stmmac_priv *priv)
                }
                wmb();
                priv->hw->desc->set_rx_owner(p + entry);
+               wmb();
        }
 }
 

The first wmb() is bogus, since we don't need any wmb() before or after setting the own bit on fragments.

The second wmb() is performed after setting the own bit on the first desc (something that is always done last). This also seems bogus, since there already was a wmb() just before set_tx_owner(first), and a writel() is performed shortly after.

The last wmb(), after set_rx_owner(), might actually be needed, since the commit message refered to problems with RX, and I don't see any writel() being performed after this.

It is worth noting that the last barrier was changed to a dma_wmb() by Pavel in: ad688cdbb076 ("stmmac: fix memory barriers").

TL;DL:
I will send a patch that removes the barriers performed after the last descriptor write for TX, since a writel() is performed shortly after.

However for RX, since this barrier is performed after setting the own bit, and there is no writel() performed shortly after, I don't know if this should be a dma_wmb() or has to be a wmb().

Best regards,
Niklas

#10 lore raw

Re: [PATCH net-next 2/4] net: stmmac: use correct barrier between coherent memory and MMIO

In reply to David Miller
Quoted 28 lines · David Miller

From: Niklas Cassel <[email protected]>
Date: Wed, 7 Mar 2018 18:21:57 +0100

Quoted 3 lines

Considering this, you can drop/revert:
95eb930a40a0 ("net: stmmac: use correct barrier between coherent memory and MMIO") or perhaps you want me to send a revert?

You must submit explicit patches to do a revert or any other change.

Quoted 4 lines

After reverting 95eb930a40a0, we will still have a dma_wmb() _after_ the last descriptor word write. You just explained that nothing else is needed after the last descriptor word write, so I actually think that this last barrier is superfluous.

You don't need one after the last descriptor write.

Look, you're only concerned with ordering within the descriptor writes.

So it's only about:

	desc->a = x;

	/* Write to 'a' must be visible to the hardware before 'b'. */
	dma_wmb();
	desc->b = y;

	writel();
Quoted 1 line

That's all that you need.

We may need to fix the docs then, there's wmb() there in the docs:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/memory-barriers.txt?h=v4.16-rc1#n1913

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#11 lore raw

PATCH 3/4 net: stmmac: ensure that the device has released ownership before reading data

In reply to Niklas Cassel

According to Documentation/memory-barriers.txt, we need to use a dma_rmb() after reading the status/own bit, to ensure that all descriptor fields are read after reading the own bit.

This way, we ensure that the DMA engine is done with the DMA descriptor before we read the other descriptor fields, e.g. reading the tx hardware timestamp (if PTP is enabled).

---
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +5 -0
index 6dd04f237b2a..a9856a8bf8ad 100644
@@ -1844,6 +1844,11 @@ static void stmmac_tx_clean(struct stmmac_priv *priv, u32 queue)
 		if (unlikely(status & tx_dma_own))
 			break;
 
+		/* Make sure descriptor fields are read after reading
+		 * the own bit.
+		 */
+		dma_rmb();
+
 		/* Just consider the last segment and ...*/
 		if (likely(!(status & tx_not_ls))) {
 			/* ... verify the status error condition */
Signature · 1 line
2.14.2
#12 lore raw

PATCH 4/4 net: stmmac: make dwmac4_release_tx_desc() clear all descriptor fields

In reply to Niklas Cassel

Make dwmac4_release_tx_desc() clear all descriptor fields, not just TDES2 and TDES3.

I'm suspecting that TDES0 and TDES1 wasn't cleared because the DMA engine uses them to store the tx hardware timestamp (if PTP is enabled).

However, stmmac_tx_clean() calls stmmac_get_tx_hwtstamp(), which reads and saves the timestamp, before it calls release_tx_desc(), so this is not an issue.

stmmac_xmit() and stmmac_tso_xmit() both always overwrite TDES0, however, stmmac_tso_xmit() sometimes sets TDES1, and since neither stmmac_xmit() nor stmmac_tso_xmit() explicitly clears TDES1, both functions might reuse a DMA descriptor with old TDES1 data.

I haven't observed any misbehavior even though TDES1 sometimes point to an old skb, however, explicitly clearing both TDES0 and TDES1 in dwmac4_release_tx_desc() minimizes the chances of undefined behavior.

---
 drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c +2 -0
index c728ffa095de..2a6521d33e43 100644
@@ -389,6 +389,8 @@ static void dwmac4_rd_prepare_tso_tx_desc(struct dma_desc *p, int is_fs,
 
 static void dwmac4_release_tx_desc(struct dma_desc *p, int mode)
 {
+	p->des0 = 0;
+	p->des1 = 0;
 	p->des2 = 0;
 	p->des3 = 0;
 }
Signature · 1 line
2.14.2