Skip to contentSkip to conversation

net/mlx5: fix Rx CQ doorbell synchronization on aarch64

PATCH In review Phil Yang 4 participants 10 messages Sep 5, 2019 – Sep 12, 2019 Network drivers

lore
Share & tools
lore

Phil Yang

author
#0 lore raw

PATCH 1/2 net/mlx5: fix Rx CQ doorbell synchronization on aarch64

The Rx completion queue doorbell field needs to be updated after the last CQE decompressed. For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the order of these operations, so use the coherent I/O memory barrier to make sure these fields are updated in order.

Fixes 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")
---
 drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +1 -1
index 9930286..e914d01 100644
@@ -727,7 +727,7 @@ rxq_burst_v(struct mlx5_rxq_data *rxq, struct rte_mbuf **pkts, uint16_t pkts_n,
 			rxq->decompressed -= n;
 		}
 	}
-	rte_compiler_barrier();
+	rte_cio_wmb();
 	*rxq->cq_db = rte_cpu_to_be_32(rxq->cq_ci);
 	return rcvd_pkt;
 }
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH 2/2 net/mlx5: fix Tx CQ doorbell synchronization on aarch64

For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the coherent memory update be observed by I/O device. It needs the coherent I/O memory barrier to enforce the ordering of Tx completion queue doorbell operation.

Fixes da1df1ccabad ("net/mlx5: fix completion queue drain loop")
---
 drivers/net/mlx5/mlx5_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx.c +1 -1
index 4c01187..c11148b 100644
@@ -2042,7 +2042,7 @@ mlx5_tx_comp_flush(struct mlx5_txq_data *restrict txq,
 	} else {
 		return;
 	}
-	rte_compiler_barrier();
+	rte_cio_wmb();
 	*txq->cq_db = rte_cpu_to_be_32(txq->cq_ci);
 	if (likely(tail != txq->elts_tail)) {
 		mlx5_tx_free_elts(txq, tail, olx);
Signature · 1 line
2.7.4
#2 lore raw

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Hi, Phil

This point is in datapath and performance is very critical.
The rte_cio_wmb() may take a lot of CPU cycles, waiting till all previous writes become visible for all external (relating to core) agents. The Tx CQE doorbelling does not need any writes to other locations to be completed, the only concern is not to reorder/merge the writes to the same doorbell register of the same sending queue in the tx_burst() internal sending loop/subsequent calls.

As far as I know - the writes to the same location should not be reordered by any arch (may be merged if memory settings allow this, it is not critical for CQE doorbell), could you, please, explain why we need explicit hardware fence before CQE doorbell update? Do you think doorbell write might be rearranged with previously reads from the ring buffer?

WBR,
Slava

Quoted 43 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Thursday, September 5, 2019 13:55
To: Yongseok Koh <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>; Nélio Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the coherent memory update be observed by I/O device. It needs the coherent I/O memory barrier to enforce the ordering of Tx completion queue doorbell operation.

Fixes da1df1ccabad ("net/mlx5: fix completion queue drain loop")
---
 drivers/net/mlx5/mlx5_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx.c +1 -1
index 4c01187..c11148b 100644
@@ -2042,7 +2042,7 @@ mlx5_tx_comp_flush(struct mlx5_txq_data
 *restrict txq,
 	} else {
 		return;
 	}
-	rte_compiler_barrier();
+	rte_cio_wmb();
 	*txq->cq_db = rte_cpu_to_be_32(txq->cq_ci);
 	if (likely(tail != txq->elts_tail)) {

mlx5_tx_free_elts(txq, tail, olx);
--
2.7.4

#3 lore raw

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Hi, Slava

Thanks for your comments.

Quoted 12 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Thursday, September 5, 2019 8:12 PM
To: Phil Yang (Arm Technology China) <[email protected]>; [email protected]; Matan Azrad <[email protected]>; Nélio Laranjeiro <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Quoted 6 lines

Hi, Phil

This point is in datapath and performance is very critical.
The rte_cio_wmb() may take a lot of CPU cycles, waiting till all previous writes become visible for all external (relating to core) agents.
The Tx CQE doorbelling does not need any writes to other locations to be completed,

In my understanding, the PMD needs to wait till all txq fields update is completed then ring the doorbell for HW. Before the Tx CQE doorbelling, it will update the producer index of work queue in Tx queue descriptor (at line 2037). The compiler barrier cannot guarantee the ordering of these operations. So use the explicit HW fence to achieve that.

As same as the HW Tx doorbell in vectorized Tx burst routine, it uses a write memory barrier to enforce the register update visible to HW immediately. Section 32.5.2 in https://doc.dpdk.org/guides/nics/mlx5.html

Quoted 57 lines

the only concern is not to reorder/merge the writes to the same doorbell register of the same sending queue in the tx_burst() internal sending loop/subsequent calls.

As far as I know - the writes to the same location should not be reordered by any arch (may be merged if memory settings allow this, it is not critical for CQE doorbell), could you, please, explain why we need explicit hardware fence before CQE doorbell update? Do you think doorbell write might be rearranged with previously reads from the ring buffer?

WBR,
Slava

Quoted 5 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Thursday, September 5, 2019 13:55
To: Yongseok Koh <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>;

Nélio

Quoted 10 lines

Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the coherent memory update be observed by I/O device. It needs the coherent I/O memory barrier to enforce the ordering

of

Quoted 28 lines

Tx completion queue doorbell operation.

Fixes da1df1ccabad ("net/mlx5: fix completion queue drain loop")
---
 drivers/net/mlx5/mlx5_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx.c +1 -1
index 4c01187..c11148b 100644
@@ -2042,7 +2042,7 @@ mlx5_tx_comp_flush(struct mlx5_txq_data
 *restrict txq,
 	} else {
 		return;
 	}
-	rte_compiler_barrier();
+	rte_cio_wmb();
 	*txq->cq_db = rte_cpu_to_be_32(txq->cq_ci);
 	if (likely(tail != txq->elts_tail)) {

mlx5_tx_free_elts(txq, tail, olx);
--
2.7.4

#4 lore raw

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Hi, Phil

Thanks for explanations, please, see below.

Quoted 39 lines

-----Original Message-----
From: Phil Yang (Arm Technology China) <[email protected]>
Sent: Friday, September 6, 2019 10:20
To: Slava Ovsiienko <[email protected]>; Yongseok Koh <[email protected]>; Matan Azrad <[email protected]>; Nélio Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Hi, Slava

Thanks for your comments.

Quoted 5 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Thursday, September 5, 2019 8:12 PM
To: Phil Yang (Arm Technology China) <[email protected]>; [email protected]; Matan Azrad <[email protected]>; Nélio

Laranjeiro

Quoted 8 lines

<[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on
aarch64

Quoted 7 lines

Hi, Phil

This point is in datapath and performance is very critical. The rte_cio_wmb() may take a lot of CPU cycles, waiting till all previous writes become visible for all external (relating to core) agents. The Tx CQE doorbelling does not need any writes to other locations to be completed,

Quoted 4 lines

In my understanding, the PMD needs to wait till all txq fields update is completed then ring the doorbell for HW.
Before the Tx CQE doorbelling, it will update the producer index of work queue in Tx queue descriptor (at line 2037).

txq->wqe_pi is exclusively software field, not related to HW directly. We should not wait for write completions to this one (assuming the tx_burst() must be called with strict affinity settings and core can't be changed).

There may be some concern about reading from "last_cqe->wqe_counter" at the line 2037. The compiler barrier was implemented to guarantee this read is issued before doorbell write.

As for possible reordering these operations (read index from CQE at 2037 and write to CQ doorbell register at 2046):

a) read is performed from already cached area (we touched
this CQE performing ownership check very recently) so it is quite unlikely to be completed after the doorbell write

b) The only risk to read wrong data is the case of CQE overwriting by HW with CQ buffer overflow. We create the CQ ring buffer with some extra space, so completions which are "in-flight" can't overwrite the CQE is being read.

The new completion request may be issued by setting flags in WQE descriptors and following SQ doorbell write, which is already prepended by wmb. (in mlx5_tx_dbrec_cond_wmb(), line 4733). So, it seems there is no chance for CQE to be overwritten.

Quoted 12 lines

The compiler barrier cannot guarantee the ordering of these operations. So use the explicit HW fence to achieve that.

As same as the HW Tx doorbell in vectorized Tx burst routine, it uses a write memory barrier to enforce the register update visible to HW immediately. Section 32.5.2 in
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc.d
pdk.org%2Fguides%2Fnics%2Fmlx5.html&amp;data=02%7C01%7Cviacheslav o%40mellanox.com%7C76938b08a9f145c4a0dd08d7329ab932%7Ca652971 c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637033512428674501&amp;sd ata=8tdVjY0%2FHOUFo1%2BeHiuqkPadSS%2FHLeo4b97gdgEHgME%3D&am p;reserved=0

This is quite different case. PMD build descriptors (WQEs) in the memory and must guarantee these data are visible for external agents before SQ (sending queue, not completion queue) doorbelling. Now there are no vectorized Tx routines (since 19.08), but, of course, we still have the "true" write memory barrier (in mlx5_tx_dbrec_cond_wmb) for this case.

Quoted 61 lines
Quoted 3 lines

the only concern is not to reorder/merge the writes to the same doorbell register of the same sending queue in the tx_burst() internal

sending loop/subsequent calls.

Quoted 6 lines

As far as I know - the writes to the same location should not be reordered by any arch (may be merged if memory settings allow this, it is not critical for CQE doorbell), could you, please, explain why we need explicit hardware fence before CQE doorbell update? Do you think doorbell write might be rearranged with previously reads from the ring

buffer?

Quoted 50 lines

WBR,
Slava

Quoted 5 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Thursday, September 5, 2019 13:55
To: Yongseok Koh <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>;

Nélio

Quoted 11 lines

Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the coherent memory update be observed by I/O device. It needs the coherent I/O memory barrier to enforce the ordering

of

Quoted 28 lines

Tx completion queue doorbell operation.

Fixes da1df1ccabad ("net/mlx5: fix completion queue drain loop")
---
 drivers/net/mlx5/mlx5_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/net/mlx5/mlx5_rxtx.c index 4c01187..c11148b 100644

drivers/net/mlx5/mlx5_rxtx.c +1 -1
@@ -2042,7 +2042,7 @@ mlx5_tx_comp_flush(struct mlx5_txq_data
 *restrict txq,
 	} else {
 		return;
 	}
-	rte_compiler_barrier();
+	rte_cio_wmb();
 	*txq->cq_db = rte_cpu_to_be_32(txq->cq_ci);
 	if (likely(tail != txq->elts_tail)) {

mlx5_tx_free_elts(txq, tail, olx);
--
2.7.4

#5 lore raw

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Quoted 56 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Friday, September 6, 2019 8:27 PM
To: Phil Yang (Arm Technology China) <[email protected]>; [email protected]; Matan Azrad <[email protected]>; Nélio Laranjeiro <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Hi, Phil

Thanks for explanations, please, see below.

Quoted 39 lines

-----Original Message-----
From: Phil Yang (Arm Technology China) <[email protected]>
Sent: Friday, September 6, 2019 10:20
To: Slava Ovsiienko <[email protected]>; Yongseok Koh <[email protected]>; Matan Azrad <[email protected]>; Nélio Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Hi, Slava

Thanks for your comments.

Quoted 5 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Thursday, September 5, 2019 8:12 PM
To: Phil Yang (Arm Technology China) <[email protected]>; [email protected]; Matan Azrad <[email protected]>; Nélio

Laranjeiro

Quoted 15 lines

<[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on
aarch64

Hi, Phil

This point is in datapath and performance is very critical. The rte_cio_wmb() may take a lot of CPU cycles, waiting till all previous writes become visible for all external (relating to core) agents. The Tx CQE doorbelling does not need any writes to other locations to be completed,

Quoted 8 lines
Quoted 4 lines

In my understanding, the PMD needs to wait till all txq fields update is completed then ring the doorbell for HW.
Before the Tx CQE doorbelling, it will update the producer index of work queue in Tx queue descriptor (at line 2037).

txq->wqe_pi is exclusively software field, not related to HW directly. We should not wait for write completions to this one (assuming the tx_burst() must be called with strict affinity settings and core can't be changed).

Understood, I really appreciate your explanation.
Could you please review the 1/2 patch in this series? All your comments are welcomed.

Quoted 11 lines

There may be some concern about reading from "last_cqe->wqe_counter" at the line 2037. The compiler barrier was implemented to guarantee this read is issued before doorbell write.

As for possible reordering these operations (read index from CQE at 2037 and write to CQ doorbell register at 2046):

a) read is performed from already cached area (we touched
this CQE performing ownership check very recently) so it is quite unlikely to be completed after the doorbell write

Yes. The "last_cqe->we_counter " is cached. However, it might not guarantee the cached data is valid when CPU issues the read operation. Because mlx5_cqe is in the coherent memory and is shared with the HW, so updating of any other filed in CQE will invalid the whole cache line. In that case, it is possible to complete the doorbell write before the wqe_counter read completed.
So we might need a rte_cio_rmb() here, right?

Quoted 99 lines

b) The only risk to read wrong data is the case of CQE overwriting by HW with CQ buffer overflow. We create the CQ ring buffer with some extra space,
so completions which are "in-flight" can't overwrite the CQE is being read.

The new completion request may be issued by setting flags in WQE descriptors
and following SQ doorbell write, which is already prepended by wmb. (in mlx5_tx_dbrec_cond_wmb(), line 4733). So, it seems there is no chance for CQE to be overwritten.

Quoted 7 lines

The compiler barrier cannot guarantee the ordering of these operations. So use the explicit HW fence to achieve that.

As same as the HW Tx doorbell in vectorized Tx burst routine, it uses a write memory barrier to enforce the register update visible to HW immediately. Section 32.5.2 in

https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc.d

Quoted 1 line

pdk.org%2Fguides%2Fnics%2Fmlx5.html&amp;data=02%7C01%7Cviacheslav

Quoted 3 lines

o%40mellanox.com%7C76938b08a9f145c4a0dd08d7329ab932%7Ca652971 c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637033512428674501&amp;sd

ata=8tdVjY0%2FHOUFo1%2BeHiuqkPadSS%2FHLeo4b97gdgEHgME%3D&am

Quoted 1 line

p;reserved=0

This is quite different case. PMD build descriptors (WQEs) in the memory and must
guarantee these data are visible for external agents before SQ (sending queue,
not completion queue) doorbelling. Now there are no vectorized Tx routines (since 19.08),
but, of course, we still have the "true" write memory barrier (in mlx5_tx_dbrec_cond_wmb)
for this case.

Quoted 61 lines
Quoted 3 lines

the only concern is not to reorder/merge the writes to the same doorbell register of the same sending queue in the tx_burst() internal

sending loop/subsequent calls.

Quoted 6 lines

As far as I know - the writes to the same location should not be reordered by any arch (may be merged if memory settings allow this, it is not critical for CQE doorbell), could you, please, explain why we need explicit hardware fence before CQE doorbell update? Do you think doorbell write might be rearranged with previously reads from the ring

buffer?

Quoted 50 lines

WBR,
Slava

Quoted 5 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Thursday, September 5, 2019 13:55
To: Yongseok Koh <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>;

Nélio

Quoted 11 lines

Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the coherent memory update be observed by I/O device. It needs the coherent I/O memory barrier to enforce the ordering

of

Quoted 28 lines

Tx completion queue doorbell operation.

Fixes da1df1ccabad ("net/mlx5: fix completion queue drain loop")
---
 drivers/net/mlx5/mlx5_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/net/mlx5/mlx5_rxtx.c index 4c01187..c11148b 100644

drivers/net/mlx5/mlx5_rxtx.c +1 -1
@@ -2042,7 +2042,7 @@ mlx5_tx_comp_flush(struct mlx5_txq_data
 *restrict txq,
 	} else {
 		return;
 	}
-	rte_compiler_barrier();
+	rte_cio_wmb();
 	*txq->cq_db = rte_cpu_to_be_32(txq->cq_ci);
 	if (likely(tail != txq->elts_tail)) {

mlx5_tx_free_elts(txq, tail, olx);
--
2.7.4

#6 lore raw

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Quoted 79 lines

-----Original Message-----
From: Phil Yang (Arm Technology China) <[email protected]>
Sent: Monday, September 9, 2019 13:12
To: Slava Ovsiienko <[email protected]>; Yongseok Koh <[email protected]>; Matan Azrad <[email protected]>; Nélio Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; Steve Capper <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Quoted 5 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Friday, September 6, 2019 8:27 PM
To: Phil Yang (Arm Technology China) <[email protected]>; [email protected]; Matan Azrad <[email protected]>; Nélio

Laranjeiro

Quoted 4 lines

<[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd

<[email protected]>

Quoted 4 lines

Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on
aarch64

Quoted 12 lines

Hi, Phil

Thanks for explanations, please, see below.

Quoted 8 lines

-----Original Message-----
From: Phil Yang (Arm Technology China) <[email protected]>
Sent: Friday, September 6, 2019 10:20
To: Slava Ovsiienko <[email protected]>; Yongseok Koh <[email protected]>; Matan Azrad <[email protected]>; Nélio Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu

(Arm

Quoted 37 lines
Quoted 36 lines

Technology China) <[email protected]>; nd <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on
aarch64

Hi, Slava

Thanks for your comments.

Quoted 5 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Thursday, September 5, 2019 8:12 PM
To: Phil Yang (Arm Technology China) <[email protected]>; [email protected]; Matan Azrad <[email protected]>; Nélio

Laranjeiro

Quoted 8 lines

<[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>; nd <[email protected]>; [email protected]
Subject: RE: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on
aarch64

Quoted 7 lines

Hi, Phil

This point is in datapath and performance is very critical. The rte_cio_wmb() may take a lot of CPU cycles, waiting till all previous writes become visible for all external (relating to core) agents. The Tx CQE doorbelling does not need any writes to other locations to be completed,

In my understanding, the PMD needs to wait till all txq fields update is completed then ring the doorbell for HW. Before the Tx CQE doorbelling, it will update the producer index of work queue in Tx queue descriptor (at line 2037).

Quoted 8 lines
Quoted 3 lines

txq->wqe_pi is exclusively software field, not related to HW directly. We should not wait for write completions to this one (assuming the tx_burst() must be called with strict affinity settings and core can't be

changed).

Understood, I really appreciate your explanation.
Could you please review the 1/2 patch in this series? All your comments are welcomed.

I could, but there is more reliable way - I asked the more Rx datapath experienced guy to do it. If we find he is too busy, I'll review the rx related part by myself.

Quoted 12 lines
Quoted 11 lines

There may be some concern about reading from "last_cqe->wqe_counter" at the line 2037. The compiler barrier was implemented to guarantee this read is issued before doorbell write.

As for possible reordering these operations (read index from CQE at 2037 and write to CQ doorbell register at 2046):

a) read is performed from already cached area (we touched this CQE performing ownership check very recently) so it is quite unlikely to be completed after the doorbell write

Quoted 7 lines

Yes. The "last_cqe->we_counter " is cached. However, it might not guarantee the cached data is valid when CPU issues the read operation. Because mlx5_cqe is in the coherent memory and is shared with the HW, so updating of any other filed in CQE will invalid the whole cache line. In that case, it is possible to complete the doorbell write before the wqe_counter read completed.
So we might need a rte_cio_rmb() here, right?

Due to b) bullet below in my earlier reply, it does not matter whether read from CQE is reordered with write to CQ doorbell or not. CQE being read can't be overwritten (by HW) by issued completion requests (already issued to HW in sending queue, we could say these requests are "in flight" now) - completion queue is large enough to store all of them without overwriting this last CQE being read in handle completion.

New completion request (within WQEs) is issued with write to SQ doorbell, which actually is prepended with "true" rte_cio_wmb(). Actually, we could drop the compiler barrier ever, but we do not dare 😊. It is not bad to have some inexpensive (or ever free of charge) insurance from unexpected code optimization done by compiler after some refactoring (porting to new platform, etc).

WBR, Slava

Quoted 109 lines
Quoted 7 lines

b) The only risk to read wrong data is the case of CQE overwriting by HW with CQ buffer overflow. We create the CQ ring buffer with some extra space, so completions which are "in-flight" can't overwrite the CQE is being read.

Quoted 11 lines

The new completion request may be issued by setting flags in WQE descriptors and following SQ doorbell write, which is already prepended by wmb.
(in mlx5_tx_dbrec_cond_wmb(), line 4733). So, it seems there is no chance for CQE to be overwritten.

Quoted 5 lines

The compiler barrier cannot guarantee the ordering of these operations. So use the explicit HW fence to achieve that.

As same as the HW Tx doorbell in vectorized Tx burst routine, it uses a write memory barrier to enforce the register update visible to HW

immediately.

Quoted 5 lines
Quoted 2 lines

Section 32.5.2 in

https://doc.d

Quoted 1 line

pdk.org%2Fguides%2Fnics%2Fmlx5.html&amp;data=02%7C01%7Cviacheslav

Quoted 1 line
Quoted 1 line

o%40mellanox.com%7C76938b08a9f145c4a0dd08d7329ab932%7Ca652971

Quoted 1 line
Quoted 1 line

c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637033512428674501&amp;sd

Quoted 2 lines
Quoted 1 line

ata=8tdVjY0%2FHOUFo1%2BeHiuqkPadSS%2FHLeo4b97gdgEHgME%3D&am

Quoted 32 lines
Quoted 1 line

p;reserved=0

This is quite different case. PMD build descriptors (WQEs) in the memory and must guarantee these data are visible for external agents before SQ (sending queue, not completion queue) doorbelling. Now there are no vectorized Tx routines (since 19.08), but, of course, we still have the "true" write memory barrier (in mlx5_tx_dbrec_cond_wmb)
for this case.

Quoted 22 lines
Quoted 4 lines

the only concern is not to reorder/merge the writes to the same doorbell register of the same sending queue in the tx_burst() internal

sending loop/subsequent calls.

Quoted 7 lines

As far as I know - the writes to the same location should not be reordered by any arch (may be merged if memory settings allow this, it is not critical for CQE doorbell), could you, please, explain why we need explicit hardware fence before CQE doorbell update? Do you think doorbell write might be rearranged with previously reads from the ring

buffer?

Quoted 9 lines

WBR,
Slava

Quoted 5 lines

-----Original Message-----
From: Phil Yang <[email protected]>
Sent: Thursday, September 5, 2019 13:55
To: Yongseok Koh <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad

<[email protected]>;

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Nélio

Quoted 3 lines

Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
Quoted 9 lines

[email protected]
Subject: [PATCH 2/2] net/mlx5: fix Tx CQ doorbell
synchronization on
aarch64

For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the coherent memory update be observed by I/O device. It needs the coherent I/O memory barrier to enforce the ordering

of

Quoted 29 lines

Tx completion queue doorbell operation.

Fixes da1df1ccabad ("net/mlx5: fix completion queue drain

loop")
Cc: [email protected]

---
 drivers/net/mlx5/mlx5_rxtx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/net/mlx5/mlx5_rxtx.c index 4c01187..c11148b 100644

drivers/net/mlx5/mlx5_rxtx.c +1 -1
@@ -2042,7 +2042,7 @@ mlx5_tx_comp_flush(struct mlx5_txq_data
 *restrict txq,
 	} else {
 		return;
 	}
-	rte_compiler_barrier();
+	rte_cio_wmb();
 	*txq->cq_db = rte_cpu_to_be_32(txq->cq_ci);
 	if (likely(tail != txq->elts_tail)) {

mlx5_tx_free_elts(txq, tail, olx);
--
2.7.4

#7 lore raw

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix Tx CQ doorbell synchronization on aarch64

Hi, Slava

Thank you for taking the time to review this patch.
I agree with you. The current design is very comprehensive. I will abandon this patch.

Thanks,
Phil Yang

Acked-by on v1 1/2
#8 lore raw
In reply to Phil Yang

From: Phil Yang

Quoted 8 lines

Subject: [PATCH 1/2] net/mlx5: fix Rx CQ doorbell synchronization on aarch64

The Rx completion queue doorbell field needs to be updated after the last CQE decompressed. For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the order of these operations, so use the coherent I/O memory barrier to make sure these fields are updated in order.

Quoted 6 lines
Fixes 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")
#9 lore raw
In reply to Phil Yang

Hi,

Quoted 18 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Phil Yang
Sent: Thursday, September 5, 2019 1:55 PM
To: Yongseok Koh <[email protected]>; Slava Ovsiienko <[email protected]>; Matan Azrad <[email protected]>; Nélio Laranjeiro <[email protected]>; [email protected]
Cc: Thomas Monjalon <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [dpdk-dev] [PATCH 1/2] net/mlx5: fix Rx CQ doorbell synchronization on aarch64

The Rx completion queue doorbell field needs to be updated after the last CQE decompressed. For the weaker memory model processors, the compiler barrier is not sufficient to guarantee the order of these operations, so use the coherent I/O memory barrier to make sure these fields are updated in order.

Quoted 7 lines
Fixes 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")

---

Patch applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh