Skip to contentSkip to conversation

r8169: improve memory barriers

PATCH In review Heiner Kallweit 3 participants 10 messages Apr 19, 2020

Subsystems (3)
lore
Share & tools
lore

Heiner Kallweit

author maintainer
#0 lore raw

v2 cover r8169: improve memory barriers

This series includes improvements for (too heavy) memory barriers. Tested on x86-64 with RTL8168g chip version.

v2:
- drop patch 2 from the series

Heiner Kallweit (3):
r8169: use smp_store_mb in rtl_tx
r8169: replace dma_rmb with READ_ONCE in rtl_rx
r8169: use WRITE_ONCE instead of dma_wmb in rtl8169_mark_to_asic

 drivers/net/ethernet/realtek/r8169_main.c | 24 ++++++++---------------
 1 file changed, 8 insertions(+), 16 deletions(-)
Signature · 1 line
2.26.1
author maintainer
#1 lore raw

PATCH v2 1/3 r8169: use smp_store_mb in rtl_tx

Use helper smp_store_mb() instead of open-coding the functionality.

---
 drivers/net/ethernet/realtek/r8169_main.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/net/ethernet/realtek/r8169_main.c +1 -2
index 1bc415d00..dd6113fd7 100644
@@ -4438,7 +4438,6 @@ static void rtl_tx(struct net_device *dev, struct rtl8169_private *tp,
 		tp->tx_stats.bytes += bytes_compl;
 		u64_stats_update_end(&tp->tx_stats.syncp);
 
-		tp->dirty_tx = dirty_tx;
 		/* Sync with rtl8169_start_xmit:
 		 * - publish dirty_tx ring index (write barrier)
 		 * - refresh cur_tx ring index and queue status (read barrier)
@@ -4446,7 +4445,7 @@ static void rtl_tx(struct net_device *dev, struct rtl8169_private *tp,
 		 * a racing xmit thread can only have a right view of the
 		 * ring status.
 		 */
-		smp_mb();
+		smp_store_mb(tp->dirty_tx, dirty_tx);
 		if (netif_queue_stopped(dev) &&
 		    rtl_tx_slots_avail(tp, MAX_SKB_FRAGS)) {
 			netif_wake_queue(dev);
Signature · 1 line
2.26.1
author maintainer
#2 lore raw

PATCH v2 2/3 r8169: replace dma_rmb with READ_ONCE in rtl_rx

In reply to Heiner Kallweit

We want to ensure that desc->opts1 is read before desc->opts2. This doesn't require a full compiler barrier. READ_ONCE provides the ordering guarantee we need.

---
 drivers/net/ethernet/realtek/r8169_main.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)
drivers/net/ethernet/realtek/r8169_main.c +3 -8
index dd6113fd7..2fc65aca3 100644
@@ -1548,7 +1548,7 @@ static inline u32 rtl8169_tx_vlan_tag(struct sk_buff *skb)
 
 static void rtl8169_rx_vlan_tag(struct RxDesc *desc, struct sk_buff *skb)
 {
-	u32 opts2 = le32_to_cpu(desc->opts2);
+	u32 opts2 = le32_to_cpu(READ_ONCE(desc->opts2));
 
 	if (opts2 & RxVlanTag)
 		__vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), swab16(opts2 & 0xffff));
@@ -4490,16 +4490,11 @@ static int rtl_rx(struct net_device *dev, struct rtl8169_private *tp, u32 budget
 		struct RxDesc *desc = tp->RxDescArray + entry;
 		u32 status;
 
-		status = le32_to_cpu(desc->opts1);
+		/* Use READ_ONCE to order descriptor field reads */
+		status = le32_to_cpu(READ_ONCE(desc->opts1));
 		if (status & DescOwn)
 			break;
 
-		/* This barrier is needed to keep us from reading
-		 * any other fields out of the Rx descriptor until
-		 * we know the status of DescOwn
-		 */
-		dma_rmb();
-
 		if (unlikely(status & RxRES)) {
 			netif_info(tp, rx_err, dev, "Rx ERROR. status = %08x\n",
 				   status);
Signature · 1 line
2.26.1
author maintainer
#3 lore raw

PATCH v2 3/3 r8169: use WRITE_ONCE instead of dma_wmb in rtl8169_mark_to_asic

In reply to Heiner Kallweit

We want to ensure that desc->opts1 is written as last descriptor field. This doesn't require a full compiler barrier, WRITE_ONCE provides the ordering guarantee we need.

---
 drivers/net/ethernet/realtek/r8169_main.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/net/ethernet/realtek/r8169_main.c +4 -6
index 2fc65aca3..3e4ed2528 100644
@@ -3892,11 +3892,9 @@ static inline void rtl8169_mark_to_asic(struct RxDesc *desc)
 {
 	u32 eor = le32_to_cpu(desc->opts1) & RingEnd;
 
-	desc->opts2 = 0;
-	/* Force memory writes to complete before releasing descriptor */
-	dma_wmb();
-
-	desc->opts1 = cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE);
+	/* Ensure ordering of writes */
+	WRITE_ONCE(desc->opts2, 0);
+	WRITE_ONCE(desc->opts1, cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE));
 }
 
 static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
@@ -3919,7 +3917,7 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
 		return NULL;
 	}
 
-	desc->addr = cpu_to_le64(mapping);
+	WRITE_ONCE(desc->addr, cpu_to_le64(mapping));
 	rtl8169_mark_to_asic(desc);
 
 	return data;
Signature · 1 line
2.26.1
#4 lore raw

Re: [PATCH net-next v2 3/3] r8169: use WRITE_ONCE instead of dma_wmb in rtl8169_mark_to_asic

Quoted 14 lines · Heiner Kallweit

We want to ensure that desc->opts1 is written as last descriptor field. This doesn't require a full compiler barrier, WRITE_ONCE provides the ordering guarantee we need.

---
 drivers/net/ethernet/realtek/r8169_main.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/net/ethernet/realtek/r8169_main.c +0 -0
index 2fc65aca3..3e4ed2528 100644

drivers/net/ethernet/realtek/r8169_main.c static inline void rtl8169_mark_to_asic(struct RxDesc *desc) · line 3897

Quoted 6 lines
drivers/net/ethernet/realtek/r8169_main.c +0 -3
@@ -3892,11 +3892,9 @@ static inline void rtl8169_mark_to_asic(struct RxDesc *desc)
 {
 	u32 eor = le32_to_cpu(desc->opts1) & RingEnd;
 
-	desc->opts2 = 0;
-	/* Force memory writes to complete before releasing descriptor */
-	dma_wmb();

If dma_wmb() was really ever needed here you should leave it even after you order these writes with WRITE_ONCE(). If not, then good riddance.

Just saying, i am not familiar with the hardware nor with the driver. :)

Petko

Quoted 21 lines
drivers/net/ethernet/realtek/r8169_main.c +4 -3
-
-	desc->opts1 = cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE);
+	/* Ensure ordering of writes */
+	WRITE_ONCE(desc->opts2, 0);
+	WRITE_ONCE(desc->opts1, cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE));
 }
 
 static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
@@ -3919,7 +3917,7 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
 		return NULL;
 	}
 
-	desc->addr = cpu_to_le64(mapping);
+	WRITE_ONCE(desc->addr, cpu_to_le64(mapping));
 	rtl8169_mark_to_asic(desc);
 
 	return data;
Signature · 1 line
2.26.1
author maintainer
#5 lore raw

Re: [PATCH net-next v2 3/3] r8169: use WRITE_ONCE instead of dma_wmb in rtl8169_mark_to_asic

Quoted 18 lines · Petko Manolov
Quoted 17 lines · Heiner Kallweit

We want to ensure that desc->opts1 is written as last descriptor field. This doesn't require a full compiler barrier, WRITE_ONCE provides the ordering guarantee we need.

---
 drivers/net/ethernet/realtek/r8169_main.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/net/ethernet/realtek/r8169_main.c +0 -0
index 2fc65aca3..3e4ed2528 100644
@@ -3892,11 +3892,9 @@ static inline void rtl8169_mark_to_asic(struct RxDesc *desc)
 {
 	u32 eor = le32_to_cpu(desc->opts1) & RingEnd;
 
Quoted 7 lines

drivers/net/ethernet/realtek/r8169_main.c static inline void rtl8169_mark_to_asic(struct RxDesc *desc) · line 3897

Quoted 3 lines
drivers/net/ethernet/realtek/r8169_main.c +0 -3
-	desc->opts2 = 0;
-	/* Force memory writes to complete before releasing descriptor */
-	dma_wmb();

If dma_wmb() was really ever needed here you should leave it even after you order these writes with WRITE_ONCE(). If not, then good riddance.

My understanding is that we have to avoid transferring ownership of descriptor to device (by setting DescOwn bit) before opts2 field has been written. Using WRITE_ONCE() should be sufficient to prevent the compiler from merging or reordering the writes.
At least that's how I read the process_level() example in
https://www.kernel.org/doc/Documentation/memory-barriers.txt

Quoted 27 lines

Just saying, i am not familiar with the hardware nor with the driver. :)

Petko

Quoted 21 lines
drivers/net/ethernet/realtek/r8169_main.c +4 -3
-
-	desc->opts1 = cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE);
+	/* Ensure ordering of writes */
+	WRITE_ONCE(desc->opts2, 0);
+	WRITE_ONCE(desc->opts1, cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE));
 }
 
 static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
@@ -3919,7 +3917,7 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
 		return NULL;
 	}
 
-	desc->addr = cpu_to_le64(mapping);
+	WRITE_ONCE(desc->addr, cpu_to_le64(mapping));
 	rtl8169_mark_to_asic(desc);
 
 	return data;
Signature · 1 line
2.26.1
#6 lore raw

Re: [PATCH net-next v2 3/3] r8169: use WRITE_ONCE instead of dma_wmb in rtl8169_mark_to_asic

Quoted 26 lines · Heiner Kallweit
Quoted 18 lines · Petko Manolov
Quoted 17 lines · Heiner Kallweit

We want to ensure that desc->opts1 is written as last descriptor field. This doesn't require a full compiler barrier, WRITE_ONCE provides the ordering guarantee we need.

---
 drivers/net/ethernet/realtek/r8169_main.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/net/ethernet/realtek/r8169_main.c +0 -0
index 2fc65aca3..3e4ed2528 100644
@@ -3892,11 +3892,9 @@ static inline void rtl8169_mark_to_asic(struct RxDesc *desc)
 {
 	u32 eor = le32_to_cpu(desc->opts1) & RingEnd;
 
Quoted 7 lines

drivers/net/ethernet/realtek/r8169_main.c static inline void rtl8169_mark_to_asic(struct RxDesc *desc) · line 3897

Quoted 3 lines
drivers/net/ethernet/realtek/r8169_main.c +0 -3
-	desc->opts2 = 0;
-	/* Force memory writes to complete before releasing descriptor */
-	dma_wmb();

If dma_wmb() was really ever needed here you should leave it even after you order these writes with WRITE_ONCE(). If not, then good riddance.

Quoted 6 lines

My understanding is that we have to avoid transferring ownership of descriptor to device (by setting DescOwn bit) before opts2 field has been written. Using WRITE_ONCE() should be sufficient to prevent the compiler from merging or reordering the writes.
At least that's how I read the process_level() example in
https://www.kernel.org/doc/Documentation/memory-barriers.txt

WRITE_ONCE() will prevent the compiler from reordering, but not the CPU. On x86 this code will run just fine, but not on ARM or PPC. Based on your description above i think dma_wmb() is still needed.

Petko

Quoted 28 lines
Quoted 27 lines

Just saying, i am not familiar with the hardware nor with the driver. :)

Petko

Quoted 21 lines
drivers/net/ethernet/realtek/r8169_main.c +4 -3
-
-	desc->opts1 = cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE);
+	/* Ensure ordering of writes */
+	WRITE_ONCE(desc->opts2, 0);
+	WRITE_ONCE(desc->opts1, cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE));
 }
 
 static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
@@ -3919,7 +3917,7 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
 		return NULL;
 	}
 
-	desc->addr = cpu_to_le64(mapping);
+	WRITE_ONCE(desc->addr, cpu_to_le64(mapping));
 	rtl8169_mark_to_asic(desc);
 
 	return data;
Signature · 1 line
2.26.1
author maintainer
#7 lore raw

Re: [PATCH net-next v2 3/3] r8169: use WRITE_ONCE instead of dma_wmb in rtl8169_mark_to_asic

Quoted 34 lines · Petko Manolov
Quoted 26 lines · Heiner Kallweit
Quoted 18 lines · Petko Manolov
Quoted 17 lines · Heiner Kallweit

We want to ensure that desc->opts1 is written as last descriptor field. This doesn't require a full compiler barrier, WRITE_ONCE provides the ordering guarantee we need.

---
 drivers/net/ethernet/realtek/r8169_main.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/net/ethernet/realtek/r8169_main.c +0 -0
index 2fc65aca3..3e4ed2528 100644
@@ -3892,11 +3892,9 @@ static inline void rtl8169_mark_to_asic(struct RxDesc *desc)
 {
 	u32 eor = le32_to_cpu(desc->opts1) & RingEnd;
 
Quoted 7 lines

drivers/net/ethernet/realtek/r8169_main.c static inline void rtl8169_mark_to_asic(struct RxDesc *desc) · line 3897

Quoted 3 lines
drivers/net/ethernet/realtek/r8169_main.c +0 -3
-	desc->opts2 = 0;
-	/* Force memory writes to complete before releasing descriptor */
-	dma_wmb();

If dma_wmb() was really ever needed here you should leave it even after you order these writes with WRITE_ONCE(). If not, then good riddance.

Quoted 6 lines

My understanding is that we have to avoid transferring ownership of descriptor to device (by setting DescOwn bit) before opts2 field has been written. Using WRITE_ONCE() should be sufficient to prevent the compiler from merging or reordering the writes.
At least that's how I read the process_level() example in
https://www.kernel.org/doc/Documentation/memory-barriers.txt

Quoted 4 lines

WRITE_ONCE() will prevent the compiler from reordering, but not the CPU. On x86 this code will run just fine, but not on ARM or PPC. Based on your description above i think dma_wmb() is still needed.

After reading a little about ARMv8-A and weakly-ordered memory I tend to agree with you. It's a pity that I don't know of any use of Realtek network chips on such a weakly-ordered platform (what of course doesn't mean it doesn't exist). This would really help me in testing and avoid such "but it works on x86" ..

Quoted 32 lines

Petko

Quoted 28 lines
Quoted 27 lines

Just saying, i am not familiar with the hardware nor with the driver. :)

Petko

Quoted 21 lines
drivers/net/ethernet/realtek/r8169_main.c +4 -3
-
-	desc->opts1 = cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE);
+	/* Ensure ordering of writes */
+	WRITE_ONCE(desc->opts2, 0);
+	WRITE_ONCE(desc->opts1, cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE));
 }
 
 static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
@@ -3919,7 +3917,7 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
 		return NULL;
 	}
 
-	desc->addr = cpu_to_le64(mapping);
+	WRITE_ONCE(desc->addr, cpu_to_le64(mapping));
 	rtl8169_mark_to_asic(desc);
 
 	return data;
Signature · 1 line
2.26.1
#8 lore raw

Re: [PATCH net-next v2 3/3] r8169: use WRITE_ONCE instead of dma_wmb in rtl8169_mark_to_asic

In reply to Heiner Kallweit
Quoted 36 lines · Heiner Kallweit
Quoted 18 lines · Petko Manolov
Quoted 17 lines · Heiner Kallweit

We want to ensure that desc->opts1 is written as last descriptor field. This doesn't require a full compiler barrier, WRITE_ONCE provides the ordering guarantee we need.

---
 drivers/net/ethernet/realtek/r8169_main.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/net/ethernet/realtek/r8169_main.c +0 -0
index 2fc65aca3..3e4ed2528 100644
@@ -3892,11 +3892,9 @@ static inline void rtl8169_mark_to_asic(struct RxDesc *desc)
 {
 	u32 eor = le32_to_cpu(desc->opts1) & RingEnd;
 
Quoted 7 lines

drivers/net/ethernet/realtek/r8169_main.c static inline void rtl8169_mark_to_asic(struct RxDesc *desc) · line 3897

Quoted 3 lines
drivers/net/ethernet/realtek/r8169_main.c +0 -3
-	desc->opts2 = 0;
-	/* Force memory writes to complete before releasing descriptor */
-	dma_wmb();

If dma_wmb() was really ever needed here you should leave it even after you order these writes with WRITE_ONCE(). If not, then good riddance.

My understanding is that we have to avoid transferring ownership of descriptor to device (by setting DescOwn bit) before opts2 field has been written. Using WRITE_ONCE() should be sufficient to prevent the compiler from merging or reordering the writes.
At least that's how I read the process_level() example in
https://www.kernel.org/doc/Documentation/memory-barriers.txt

Quoted 3 lines

Just saying, i am not familiar with the hardware nor with the driver. :)

Quoted 8 lines
Quoted 8 lines

Petko

drivers/net/ethernet/realtek/r8169_main.c static inline void rtl8169_mark_to_asic(struct RxDesc *desc) · line 3897

Quoted 5 lines
drivers/net/ethernet/realtek/r8169_main.c +3 -2
-
-	desc->opts1 = cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE);
+	/* Ensure ordering of writes */
+	WRITE_ONCE(desc->opts2, 0);
+	WRITE_ONCE(desc->opts1, cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE));

No, this is not correct.

WRITE_ONCE(X, ...)
WRITE_ONCE(Y, ...)

has no guarantee X is written before Y

Sure, the compiler is making sure to perform one write for X, one for Y, but you want stronger semantic than that. You want to keep dma_wmb()

However, this part of your patch seems fine :

- desc->opts1 = cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE);
+ WRITE_ONCE(desc->opts1, cpu_to_le32(DescOwn | eor | R8169_RX_BUF_SIZE));

Quoted 17 lines
Quoted 16 lines
Quoted 16 lines
drivers/net/ethernet/realtek/r8169_main.c +1 -1
 }
 
 static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
@@ -3919,7 +3917,7 @@ static struct page *rtl8169_alloc_rx_data(struct rtl8169_private *tp,
 		return NULL;
 	}
 
-	desc->addr = cpu_to_le64(mapping);
+	WRITE_ONCE(desc->addr, cpu_to_le64(mapping));
 	rtl8169_mark_to_asic(desc);
 
 	return data;
Signature · 1 line
2.26.1
author maintainer
#9 lore raw
In reply to Heiner Kallweit
Quoted 3 lines · Heiner Kallweit

This series includes improvements for (too heavy) memory barriers. Tested on x86-64 with RTL8168g chip version.

A recent conversation convinced me that few of these change are not safe on platforms with weakly-ordered memory. Therefore please disregard the series.

Quoted 11 lines

v2:
- drop patch 2 from the series

Heiner Kallweit (3):
r8169: use smp_store_mb in rtl_tx
r8169: replace dma_rmb with READ_ONCE in rtl_rx
r8169: use WRITE_ONCE instead of dma_wmb in rtl8169_mark_to_asic

 drivers/net/ethernet/realtek/r8169_main.c | 24 ++++++++---------------
 1 file changed, 8 insertions(+), 16 deletions(-)