Skip to contentSkip to conversation

net/mlx5: fix risk in Rx descriptor read in NEON vector path

PATCH In review Ruifeng Wang 4 participants 13 messages Jan 4, 2022 – Jun 19, 2023 Network drivers

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net/mlx5: fix risk in Rx descriptor read in NEON vector path

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx completions")
---
 drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8 -0
index b1d16baa61..b1ec615b51 100644
@@ -647,6 +647,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq,
 		c0 = vld1q_u64((uint64_t *)(p0 + 48));
 		/* Synchronize for loading the rest of blocks. */
 		rte_io_rmb();
+		/* B.0 (CQE 3) reload lower half of the block. */
+		c3 = vld1q_lane_u64((uint64_t *)(p3 + 48), c3, 0);
+		/* B.0 (CQE 2) reload lower half of the block. */
+		c2 = vld1q_lane_u64((uint64_t *)(p2 + 48), c2, 0);
+		/* B.0 (CQE 1) reload lower half of the block. */
+		c1 = vld1q_lane_u64((uint64_t *)(p1 + 48), c1, 0);
+		/* B.0 (CQE 0) reload lower half of the block. */
+		c0 = vld1q_lane_u64((uint64_t *)(p0 + 48), c0, 0);
 		/* Prefetch next 4 CQEs. */
 		if (pkts_n - pos >= 2 * MLX5_VPMD_DESCS_PER_LOOP) {
 			unsigned int next = pos + MLX5_VPMD_DESCS_PER_LOOP;
Signature · 1 line
2.25.1
#1 lore raw

Ping.
Please could you help to review this patch?

Thanks.
Ruifeng

Quoted 53 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 11:01 AM
To: [email protected]; [email protected]
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx

completions")
Cc: [email protected]

---
 drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8 -0
index b1d16baa61..b1ec615b51 100644
@@ -647,6 +647,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,
 volatile struct mlx5_cqe *cq,
 		c0 = vld1q_u64((uint64_t *)(p0 + 48));
 		/* Synchronize for loading the rest of blocks. */
 		rte_io_rmb();
+		/* B.0 (CQE 3) reload lower half of the block. */
+		c3 = vld1q_lane_u64((uint64_t *)(p3 + 48), c3, 0);
+		/* B.0 (CQE 2) reload lower half of the block. */
+		c2 = vld1q_lane_u64((uint64_t *)(p2 + 48), c2, 0);
+		/* B.0 (CQE 1) reload lower half of the block. */
+		c1 = vld1q_lane_u64((uint64_t *)(p1 + 48), c1, 0);
+		/* B.0 (CQE 0) reload lower half of the block. */
+		c0 = vld1q_lane_u64((uint64_t *)(p0 + 48), c0, 0);
 		/* Prefetch next 4 CQEs. */
 		if (pkts_n - pos >= 2 * MLX5_VPMD_DESCS_PER_LOOP) {

unsigned int next = pos +
MLX5_VPMD_DESCS_PER_LOOP;
--
2.25.1

#2 lore raw

Hi Ruifeng,

Patch looks reasonable, thank you.
Just curious - did you see the real issue with re-ordering in this code fragment? And, please, let us do performance check.

With best regards,
Slava

Quoted 72 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Thursday, February 10, 2022 8:25
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Ping.
Please could you help to review this patch?

Thanks.
Ruifeng

Quoted 5 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 11:01 AM
To: [email protected]; [email protected]
Cc: [email protected]; Honnappa Nagarahalli

<[email protected]>;

Quoted 1 line

[email protected]; nd <[email protected]>; Ruifeng Wang

<[email protected]>

Quoted 47 lines

Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx

completions")
Cc: [email protected]

---
 drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8 -0
index b1d16baa61..b1ec615b51 100644
@@ -647,6 +647,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,
 volatile struct mlx5_cqe *cq,
 		c0 = vld1q_u64((uint64_t *)(p0 + 48));
 		/* Synchronize for loading the rest of blocks. */
 		rte_io_rmb();
+		/* B.0 (CQE 3) reload lower half of the block. */
+		c3 = vld1q_lane_u64((uint64_t *)(p3 + 48), c3, 0);
+		/* B.0 (CQE 2) reload lower half of the block. */
+		c2 = vld1q_lane_u64((uint64_t *)(p2 + 48), c2, 0);
+		/* B.0 (CQE 1) reload lower half of the block. */
+		c1 = vld1q_lane_u64((uint64_t *)(p1 + 48), c1, 0);
+		/* B.0 (CQE 0) reload lower half of the block. */
+		c0 = vld1q_lane_u64((uint64_t *)(p0 + 48), c0, 0);
 		/* Prefetch next 4 CQEs. */
 		if (pkts_n - pos >= 2 * MLX5_VPMD_DESCS_PER_LOOP) {

unsigned int next = pos +
MLX5_VPMD_DESCS_PER_LOOP;
--
2.25.1

#3 lore raw
Quoted 11 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Thursday, February 10, 2022 4:17 PM
To: Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Quoted 1 line

Hi Ruifeng,

Hi Slava,

Quoted 4 lines

Patch looks reasonable, thank you.
Just curious - did you see the real issue with re-ordering in this code fragment?

No real issue was seen. It is analysis from architecture perspective.

Quoted 1 line

And, please, let us do performance check.

Sure. Thank you.

Quoted 79 lines

With best regards,
Slava

Quoted 6 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Thursday, February 10, 2022 8:25
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli

<[email protected]>;

Quoted 1 line

[email protected]; nd <[email protected]>; Ruifeng Wang

<[email protected]>;

Quoted 10 lines

nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Ping.
Please could you help to review this patch?

Thanks.
Ruifeng

Quoted 15 lines
Quoted 5 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 11:01 AM
To: [email protected]; [email protected]
Cc: [email protected]; Honnappa Nagarahalli

<[email protected]>;

Quoted 1 line

[email protected]; nd <[email protected]>; Ruifeng Wang

<[email protected]>

Quoted 7 lines

Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could

contain invalid data.

Quoted 40 lines
Quoted 40 lines

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx

completions")
Cc: [email protected]

---
 drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8 -0
index b1d16baa61..b1ec615b51 100644
@@ -647,6 +647,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,
 volatile struct mlx5_cqe *cq,
 		c0 = vld1q_u64((uint64_t *)(p0 + 48));
 		/* Synchronize for loading the rest of blocks. */
 		rte_io_rmb();
+		/* B.0 (CQE 3) reload lower half of the block. */
+		c3 = vld1q_lane_u64((uint64_t *)(p3 + 48), c3, 0);
+		/* B.0 (CQE 2) reload lower half of the block. */
+		c2 = vld1q_lane_u64((uint64_t *)(p2 + 48), c2, 0);
+		/* B.0 (CQE 1) reload lower half of the block. */
+		c1 = vld1q_lane_u64((uint64_t *)(p1 + 48), c1, 0);
+		/* B.0 (CQE 0) reload lower half of the block. */
+		c0 = vld1q_lane_u64((uint64_t *)(p0 + 48), c0, 0);
 		/* Prefetch next 4 CQEs. */
 		if (pkts_n - pos >= 2 * MLX5_VPMD_DESCS_PER_LOOP) {

unsigned int next = pos +
MLX5_VPMD_DESCS_PER_LOOP;
--
2.25.1

Tested-by on v1 1/1
#4 lore raw
In reply to Ruifeng Wang
Quoted 21 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 5:01 AM
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Quoted 6 lines
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx

completions")
Cc: [email protected]

---

Tested with BlueField-2 and didn't see a performance impact.

Thanks,
Ali

#5 lore raw

Hi, Ruifeng

My apologies for review delay.
As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)

How the re-reading of CQEs can resolve the issue? This wrong scenario might happen on the second read and we would run into the same issue.

In my opinion, the right solution to cover potential reordering should be:
- read CQE
- check CQE status (first 8B)
- read memory barrier
- read the rest of CQE

With best regards,
Slava

Quoted 45 lines

-----Original Message-----
From: Ali Alnubani <[email protected]>
Sent: Thursday, May 19, 2022 17:56
To: Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Quoted 22 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 5:01 AM
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Quoted 6 lines
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx

completions")
Cc: [email protected]

---

Tested with BlueField-2 and didn't see a performance impact.

Thanks,
Ali

#6 lore raw
Quoted 11 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, June 20, 2022 1:38 PM
To: Ali Alnubani <[email protected]>; Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Hi, Ruifeng

Hi Slava,

Thanks for your review.

Quoted 2 lines

My apologies for review delay.

Apologies too. I was on something else.

Quoted 5 lines

As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)

Yes, This is the problem.

Quoted 4 lines

How the re-reading of CQEs can resolve the issue?
This wrong scenario might happen on the second read and we would run into the same issue.

Here we are trying to ordering reading of a 16B vector (8B with op_own - high, and 8B without op_own - low). The first read will load 16B. The second read will load and update low 8B (no op_own).
There are 2 possible status indicated by op_own: valid, invalid.
If CQE status is invalid, no problem, it will be ignored this time.
If CQE status is valid, the second read ensures the rest of CQE is no older than high 8B (with op_own). Assuming NIC updates op_own no earlier than the rest part of CQE, I think the second read ensures CQE content retrieved is correct.

Quoted 4 lines

In my opinion, the right solution to cover potential reordering should be:
- read CQE
- check CQE status (first 8B)

We don't need to check CQE status at the moment. See explanation above.

Quoted 51 lines

- read memory barrier
- read the rest of CQE

With best regards,
Slava

Quoted 45 lines

-----Original Message-----
From: Ali Alnubani <[email protected]>
Sent: Thursday, May 19, 2022 17:56
To: Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Quoted 22 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 5:01 AM
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Quoted 6 lines
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx

completions")
Cc: [email protected]

---

Tested with BlueField-2 and didn't see a performance impact.

Thanks,
Ali

#7 lore raw

Hi, Ruifeng

Quoted 38 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Monday, June 27, 2022 14:08
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Quoted 11 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, June 20, 2022 1:38 PM
To: Ali Alnubani <[email protected]>; Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Hi, Ruifeng

Hi Slava,

Thanks for your review.

Quoted 2 lines

My apologies for review delay.

Apologies too. I was on something else.

Quoted 5 lines

As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)

Yes, This is the problem.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

How the re-reading of CQEs can resolve the issue?
This wrong scenario might happen on the second read and we would run into the same issue.

Here we are trying to ordering reading of a 16B vector (8B with op_own - high, and 8B without op_own - low).
The first read will load 16B. The second read will load and update low 8B (no op_own).

OK, I got the point, thank you for the explanations.
Can we avoid the first reading of low 8B (no containing CQE owning field)?

I mean to update this part to read only upper 8Bs:

                /* B.0 (CQE 3) load a block having op_own. */
                c3 = vld1q_u64((uint64_t *)(p3 + 48));
                /* B.0 (CQE 2) load a block having op_own. */
                c2 = vld1q_u64((uint64_t *)(p2 + 48));
                /* B.0 (CQE 1) load a block having op_own. */
                c1 = vld1q_u64((uint64_t *)(p1 + 48));
                /* B.0 (CQE 0) load a block having op_own. */
                c0 = vld1q_u64((uint64_t *)(p0 + 48));
                /* Synchronize for loading the rest of blocks. */
                rte_io_rmb();

Because lower 8Bs will be overlapped with the second read (in your patch) and barrier ensures the correct order.

With best regards,
Slava

#8 lore raw
Quoted 75 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Wednesday, June 29, 2022 3:55 PM
To: Ruifeng Wang <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Hi, Ruifeng

Quoted 6 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Monday, June 27, 2022 14:08
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli

<[email protected]>;

Quoted 32 lines

[email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Quoted 11 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, June 20, 2022 1:38 PM
To: Ali Alnubani <[email protected]>; Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Hi, Ruifeng

Hi Slava,

Thanks for your review.

Quoted 2 lines

My apologies for review delay.

Apologies too. I was on something else.

Quoted 5 lines

As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)

Yes, This is the problem.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

How the re-reading of CQEs can resolve the issue?
This wrong scenario might happen on the second read and we would run into the same issue.

Here we are trying to ordering reading of a 16B vector (8B with op_own
- high, and 8B without op_own - low).
The first read will load 16B. The second read will load and update low 8B (no op_own).

OK, I got the point, thank you for the explanations.
Can we avoid the first reading of low 8B (no containing CQE owning field)?

I mean to update this part to read only upper 8Bs:

                /* B.0 (CQE 3) load a block having op_own. */
                c3 = vld1q_u64((uint64_t *)(p3 + 48));
                /* B.0 (CQE 2) load a block having op_own. */
                c2 = vld1q_u64((uint64_t *)(p2 + 48));
                /* B.0 (CQE 1) load a block having op_own. */
                c1 = vld1q_u64((uint64_t *)(p1 + 48));
                /* B.0 (CQE 0) load a block having op_own. */
                c0 = vld1q_u64((uint64_t *)(p0 + 48));
                /* Synchronize for loading the rest of blocks. */
                rte_io_rmb();
Quoted 2 lines

Because lower 8Bs will be overlapped with the second read (in your patch) and barrier ensures the correct order.

Hi Slava,

Yes, your suggestion is valid.
Actually, I tried that approach: load higher 8B + barrier + load lower 8B + combine the two 8Bs into a vector. It also has no observable performance impact but generates more instructions compared to the current patch (the 'combine' operation). So I followed current approach.

Thanks.

Quoted 4 lines

With best regards,
Slava

#9 lore raw
Quoted 89 lines

-----Original Message-----
From: Ruifeng Wang
Sent: Wednesday, June 29, 2022 7:41 PM
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Quoted 75 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Wednesday, June 29, 2022 3:55 PM
To: Ruifeng Wang <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Hi, Ruifeng

Quoted 6 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Monday, June 27, 2022 14:08
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli

<[email protected]>;

Quoted 36 lines

[email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Quoted 11 lines

-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, June 20, 2022 1:38 PM
To: Ali Alnubani <[email protected]>; Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path

Hi, Ruifeng

Hi Slava,

Thanks for your review.

Quoted 2 lines

My apologies for review delay.

Apologies too. I was on something else.

Quoted 5 lines

As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)

Yes, This is the problem.

Quoted 4 lines

How the re-reading of CQEs can resolve the issue?
This wrong scenario might happen on the second read and we would run into the same issue.

Quoted 5 lines

Here we are trying to ordering reading of a 16B vector (8B with op_own
- high, and 8B without op_own - low).
The first read will load 16B. The second read will load and update low 8B (no op_own).

OK, I got the point, thank you for the explanations.
Can we avoid the first reading of low 8B (no containing CQE owning field)?

I mean to update this part to read only upper 8Bs:

                /* B.0 (CQE 3) load a block having op_own. */
                c3 = vld1q_u64((uint64_t *)(p3 + 48));
                /* B.0 (CQE 2) load a block having op_own. */
                c2 = vld1q_u64((uint64_t *)(p2 + 48));
                /* B.0 (CQE 1) load a block having op_own. */
                c1 = vld1q_u64((uint64_t *)(p1 + 48));
                /* B.0 (CQE 0) load a block having op_own. */
                c0 = vld1q_u64((uint64_t *)(p0 + 48));
                /* Synchronize for loading the rest of blocks. */
                rte_io_rmb();
Quoted 2 lines

Because lower 8Bs will be overlapped with the second read (in your patch) and barrier ensures the correct order.

Hi Slava,

Quoted 9 lines

Yes, your suggestion is valid.
Actually, I tried that approach: load higher 8B + barrier + load lower 8B + combine the two 8Bs into a vector.
It also has no observable performance impact but generates more instructions compared to the current patch (the 'combine' operation).
So I followed current approach.

Thanks.

Quoted 1 line

Hi Slava,

Are there any further comments?

Thanks,
Ruifeng

#10 lore raw
Quoted 12 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: четверг, 29 сентября 2022 г. 09:51
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path
Hi Slava,

Quoted 2 lines

Are there any further comments?

Hi, Ruifeng

I've recalled the context and re-reviewed the patch. There is no performance impact and I'm run out of objections 😊 My sincere apologizes for the long delay ☹

#11 lore raw

PATCH v2 1/1 net/mlx5: fix risk in Rx descriptor read in NEON vector path

In reply to Ruifeng Wang

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx completions")

---
v2: Rebased and added tags that received.

 drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8 -0
index 75e8ed7e5a..9079da65de 100644
@@ -675,6 +675,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq,
 		c0 = vld1q_u64((uint64_t *)(p0 + 48));
 		/* Synchronize for loading the rest of blocks. */
 		rte_io_rmb();
+		/* B.0 (CQE 3) reload lower half of the block. */
+		c3 = vld1q_lane_u64((uint64_t *)(p3 + 48), c3, 0);
+		/* B.0 (CQE 2) reload lower half of the block. */
+		c2 = vld1q_lane_u64((uint64_t *)(p2 + 48), c2, 0);
+		/* B.0 (CQE 1) reload lower half of the block. */
+		c1 = vld1q_lane_u64((uint64_t *)(p1 + 48), c1, 0);
+		/* B.0 (CQE 0) reload lower half of the block. */
+		c0 = vld1q_lane_u64((uint64_t *)(p0 + 48), c0, 0);
 		/* Prefetch next 4 CQEs. */
 		if (pkts_n - pos >= 2 * MLX5_VPMD_DESCS_PER_LOOP) {
 			unsigned int next = pos + MLX5_VPMD_DESCS_PER_LOOP;
Signature · 1 line
2.25.1
#12 lore raw

Hi,

Quoted 23 lines

-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, May 30, 2023 8:48 AM
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>; Ali Alnubani <[email protected]>
Subject: [PATCH v2] net/mlx5: fix risk in Rx descriptor read in NEON vector path

In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.

Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.

Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.

Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx
Quoted 9 lines

completions")
Cc: [email protected]

---
v2: Rebased and added tags that received.

Patch applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh