net/mlx5: fix risk in Rx descriptor read in NEON vector path
PATCH 1/1 net/mlx5: fix risk in Rx descriptor read in NEON vector path
In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.
Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8 -0
@@ -647,6 +647,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, c0 =vld1q_u64 ((uint64_t *)(p0 +48 ));/* Synchronize for loading the rest of blocks. */ rte_io_rmb (); +/* B.0 (CQE 3) reload lower half of the block. */ + c3 =vld1q_lane_u64 ((uint64_t *)(p3 +48 ), c3,0 ); +/* B.0 (CQE 2) reload lower half of the block. */ + c2 =vld1q_lane_u64 ((uint64_t *)(p2 +48 ), c2,0 ); +/* B.0 (CQE 1) reload lower half of the block. */ + c1 =vld1q_lane_u64 ((uint64_t *)(p1 +48 ), c1,0 ); +/* B.0 (CQE 0) reload lower half of the block. */ + c0 =vld1q_lane_u64 ((uint64_t *)(p0 +48 ), c0,0 );/* Prefetch next 4 CQEs. */ if (pkts_n - pos >=2 *MLX5_VPMD_DESCS_PER_LOOP ) {unsigned int next = pos +MLX5_VPMD_DESCS_PER_LOOP ;
Signature · 1 line
2.25.1
Ping.
Please could you help to review this patch?
Thanks.
Ruifeng
Quoted 53 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 11:01 AM
To: [email protected]; [email protected]
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathIn NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes
op_ownfield could be reordered after read of other words. In this case, some words could contain invalid data.Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rxcompletions")
Cc: [email protected]Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8
-0@@ -647,6 +647,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,volatile struct mlx5_cqe *cq, c0 =vld1q_u64 ((uint64_t *)(p0 +48 ));/* Synchronize for loading the rest of blocks. */ rte_io_rmb (); +/* B.0 (CQE 3) reload lower half of the block. */ + c3 =vld1q_lane_u64 ((uint64_t *)(p3 +48 ), c3,0 ); +/* B.0 (CQE 2) reload lower half of the block. */ + c2 =vld1q_lane_u64 ((uint64_t *)(p2 +48 ), c2,0 ); +/* B.0 (CQE 1) reload lower half of the block. */ + c1 =vld1q_lane_u64 ((uint64_t *)(p1 +48 ), c1,0 ); +/* B.0 (CQE 0) reload lower half of the block. */ + c0 =vld1q_lane_u64 ((uint64_t *)(p0 +48 ), c0,0 );/* Prefetch next 4 CQEs. */ if (pkts_n - pos >=2 *MLX5_VPMD_DESCS_PER_LOOP ) {unsigned int next = pos +
MLX5_VPMD_DESCS_PER_LOOP;
--
2.25.1
Hi Ruifeng,
Patch looks reasonable, thank you.
Just curious - did you see the real issue with re-ordering in this code fragment? And, please, let us do performance check.
With best regards,
Slava
Quoted 72 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Thursday, February 10, 2022 8:25
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathPing.
Please could you help to review this patch?Thanks.
RuifengQuoted 5 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 11:01 AM
To: [email protected]; [email protected]
Cc: [email protected]; Honnappa NagarahalliQuoted 1 line
[email protected]; nd <[email protected]>; Ruifeng Wang
Quoted 47 lines
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path
In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes
op_ownfield could be reordered after read of other words. In this case, some words could contain invalid data.Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rxcompletions")
Cc: [email protected]Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8
-0@@ -647,6 +647,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,volatile struct mlx5_cqe *cq, c0 =vld1q_u64 ((uint64_t *)(p0 +48 ));/* Synchronize for loading the rest of blocks. */ rte_io_rmb (); +/* B.0 (CQE 3) reload lower half of the block. */ + c3 =vld1q_lane_u64 ((uint64_t *)(p3 +48 ), c3,0 ); +/* B.0 (CQE 2) reload lower half of the block. */ + c2 =vld1q_lane_u64 ((uint64_t *)(p2 +48 ), c2,0 ); +/* B.0 (CQE 1) reload lower half of the block. */ + c1 =vld1q_lane_u64 ((uint64_t *)(p1 +48 ), c1,0 ); +/* B.0 (CQE 0) reload lower half of the block. */ + c0 =vld1q_lane_u64 ((uint64_t *)(p0 +48 ), c0,0 );/* Prefetch next 4 CQEs. */ if (pkts_n - pos >=2 *MLX5_VPMD_DESCS_PER_LOOP ) {unsigned int next = pos +
MLX5_VPMD_DESCS_PER_LOOP;
--
2.25.1
Quoted 11 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Thursday, February 10, 2022 4:17 PM
To: Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path
Quoted 1 line
Hi Ruifeng,
Hi Slava,
Quoted 4 lines
Patch looks reasonable, thank you.
Just curious - did you see the real issue with re-ordering in this code fragment?
No real issue was seen. It is analysis from architecture perspective.
Quoted 1 line
And, please, let us do performance check.
Sure. Thank you.
Quoted 79 lines
With best regards,
SlavaQuoted 6 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Thursday, February 10, 2022 8:25
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; Honnappa NagarahalliQuoted 1 line
[email protected]; nd <[email protected]>; Ruifeng Wang
Quoted 10 lines
nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathPing.
Please could you help to review this patch?Thanks.
RuifengQuoted 15 lines
Quoted 5 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 11:01 AM
To: [email protected]; [email protected]
Cc: [email protected]; Honnappa NagarahalliQuoted 1 line
[email protected]; nd <[email protected]>; Ruifeng Wang
Quoted 7 lines
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path
In NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes
op_ownfield could be reordered after read of other words. In this case, some words couldcontain invalid data.
Quoted 40 lines
Quoted 40 lines
Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rxcompletions")
Cc: [email protected]Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++ 1 file changed, 8 insertions(+)drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8
-0@@ -647,6 +647,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,volatile struct mlx5_cqe *cq, c0 =vld1q_u64 ((uint64_t *)(p0 +48 ));/* Synchronize for loading the rest of blocks. */ rte_io_rmb (); +/* B.0 (CQE 3) reload lower half of the block. */ + c3 =vld1q_lane_u64 ((uint64_t *)(p3 +48 ), c3,0 ); +/* B.0 (CQE 2) reload lower half of the block. */ + c2 =vld1q_lane_u64 ((uint64_t *)(p2 +48 ), c2,0 ); +/* B.0 (CQE 1) reload lower half of the block. */ + c1 =vld1q_lane_u64 ((uint64_t *)(p1 +48 ), c1,0 ); +/* B.0 (CQE 0) reload lower half of the block. */ + c0 =vld1q_lane_u64 ((uint64_t *)(p0 +48 ), c0,0 );/* Prefetch next 4 CQEs. */ if (pkts_n - pos >=2 *MLX5_VPMD_DESCS_PER_LOOP ) {unsigned int next = pos +
MLX5_VPMD_DESCS_PER_LOOP;
--
2.25.1
Quoted 21 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 5:01 AM
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathIn NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes
op_ownfield could be reordered after read of other words. In this case, some words could contain invalid data.Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
Quoted 6 lines
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rxcompletions")
Cc: [email protected]Signed-off-by Ruifeng Wang <[email protected]>---
Tested with BlueField-2 and didn't see a performance impact.
Thanks,
Ali
Hi, Ruifeng
My apologies for review delay.
As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)
How the re-reading of CQEs can resolve the issue? This wrong scenario might happen on the second read and we would run into the same issue.
In my opinion, the right solution to cover potential reordering should be:
- read CQE
- check CQE status (first 8B)
- read memory barrier
- read the rest of CQE
With best regards,
Slava
Quoted 45 lines
-----Original Message-----
From: Ali Alnubani <[email protected]>
Sent: Thursday, May 19, 2022 17:56
To: Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathQuoted 22 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 5:01 AM
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathIn NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes
op_ownfield could be reordered after read of other words. In this case, some words could contain invalid data.Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
Quoted 6 lines
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rxcompletions")
Cc: [email protected]Signed-off-by Ruifeng Wang <[email protected]>---
Tested with BlueField-2 and didn't see a performance impact.
Tested-by Ali Alnubani <[email protected]>Thanks,
Ali
Quoted 11 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, June 20, 2022 1:38 PM
To: Ali Alnubani <[email protected]>; Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathHi, Ruifeng
Hi Slava,
Thanks for your review.
Quoted 2 lines
My apologies for review delay.
Apologies too. I was on something else.
Quoted 5 lines
As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)
Yes, This is the problem.
Quoted 4 lines
How the re-reading of CQEs can resolve the issue?
This wrong scenario might happen on the second read and we would run into the same issue.
Here we are trying to ordering reading of a 16B vector (8B with op_own - high, and 8B without op_own - low). The first read will load 16B. The second read will load and update low 8B (no op_own).
There are 2 possible status indicated by op_own: valid, invalid.
If CQE status is invalid, no problem, it will be ignored this time.
If CQE status is valid, the second read ensures the rest of CQE is no older than high 8B (with op_own). Assuming NIC updates op_own no earlier than the rest part of CQE, I think the second read ensures CQE content retrieved is correct.
Quoted 4 lines
In my opinion, the right solution to cover potential reordering should be:
- read CQE
- check CQE status (first 8B)
We don't need to check CQE status at the moment. See explanation above.
Quoted 51 lines
- read memory barrier
- read the rest of CQEWith best regards,
SlavaQuoted 45 lines
-----Original Message-----
From: Ali Alnubani <[email protected]>
Sent: Thursday, May 19, 2022 17:56
To: Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathQuoted 22 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, January 4, 2022 5:01 AM
To: Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathIn NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes
op_ownfield could be reordered after read of other words. In this case, some words could contain invalid data.Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
Quoted 6 lines
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rxcompletions")
Cc: [email protected]Signed-off-by Ruifeng Wang <[email protected]>---
Tested with BlueField-2 and didn't see a performance impact.
Tested-by Ali Alnubani <[email protected]>Thanks,
Ali
Hi, Ruifeng
Quoted 38 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Monday, June 27, 2022 14:08
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathQuoted 11 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, June 20, 2022 1:38 PM
To: Ali Alnubani <[email protected]>; Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathHi, Ruifeng
Hi Slava,
Thanks for your review.
Quoted 2 lines
My apologies for review delay.
Apologies too. I was on something else.
Quoted 5 lines
As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)Yes, This is the problem.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
How the re-reading of CQEs can resolve the issue?
This wrong scenario might happen on the second read and we would run into the same issue.Here we are trying to ordering reading of a 16B vector (8B with
op_own- high, and 8B withoutop_own- low).
The first read will load 16B. The second read will load and update low 8B (noop_own).
OK, I got the point, thank you for the explanations.
Can we avoid the first reading of low 8B (no containing CQE owning field)?
I mean to update this part to read only upper 8Bs:
/* B.0 (CQE 3) load a block having op_own. */
c3 = vld1q_u64((uint64_t *)(p3 + 48));
/* B.0 (CQE 2) load a block having op_own. */
c2 = vld1q_u64((uint64_t *)(p2 + 48));
/* B.0 (CQE 1) load a block having op_own. */
c1 = vld1q_u64((uint64_t *)(p1 + 48));
/* B.0 (CQE 0) load a block having op_own. */
c0 = vld1q_u64((uint64_t *)(p0 + 48));
/* Synchronize for loading the rest of blocks. */
rte_io_rmb();Because lower 8Bs will be overlapped with the second read (in your patch) and barrier ensures the correct order.
With best regards,
Slava
Quoted 75 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Wednesday, June 29, 2022 3:55 PM
To: Ruifeng Wang <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathHi, Ruifeng
Quoted 6 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Monday, June 27, 2022 14:08
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa NagarahalliQuoted 32 lines
[email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathQuoted 11 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, June 20, 2022 1:38 PM
To: Ali Alnubani <[email protected]>; Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathHi, Ruifeng
Hi Slava,
Thanks for your review.
Quoted 2 lines
My apologies for review delay.
Apologies too. I was on something else.
Quoted 5 lines
As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)Yes, This is the problem.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
How the re-reading of CQEs can resolve the issue?
This wrong scenario might happen on the second read and we would run into the same issue.Here we are trying to ordering reading of a 16B vector (8B with
op_own
- high, and 8B withoutop_own- low).
The first read will load 16B. The second read will load and update low 8B (noop_own).OK, I got the point, thank you for the explanations.
Can we avoid the first reading of low 8B (no containing CQE owning field)?I mean to update this part to read only upper 8Bs:
/* B.0 (CQE 3) load a block having op_own. */ c3 = vld1q_u64((uint64_t *)(p3 + 48)); /* B.0 (CQE 2) load a block having op_own. */ c2 = vld1q_u64((uint64_t *)(p2 + 48)); /* B.0 (CQE 1) load a block having op_own. */ c1 = vld1q_u64((uint64_t *)(p1 + 48)); /* B.0 (CQE 0) load a block having op_own. */ c0 = vld1q_u64((uint64_t *)(p0 + 48)); /* Synchronize for loading the rest of blocks. */ rte_io_rmb();
Quoted 2 lines
Because lower 8Bs will be overlapped with the second read (in your patch) and barrier ensures the correct order.
Hi Slava,
Yes, your suggestion is valid.
Actually, I tried that approach: load higher 8B + barrier + load lower 8B + combine the two 8Bs into a vector. It also has no observable performance impact but generates more instructions compared to the current patch (the 'combine' operation). So I followed current approach.
Thanks.
Quoted 4 lines
With best regards,
Slava
Quoted 89 lines
-----Original Message-----
From: Ruifeng Wang
Sent: Wednesday, June 29, 2022 7:41 PM
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathQuoted 75 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Wednesday, June 29, 2022 3:55 PM
To: Ruifeng Wang <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathHi, Ruifeng
Quoted 6 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Monday, June 27, 2022 14:08
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa NagarahalliQuoted 36 lines
[email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathQuoted 11 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, June 20, 2022 1:38 PM
To: Ali Alnubani <[email protected]>; Ruifeng Wang <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector pathHi, Ruifeng
Hi Slava,
Thanks for your review.
Quoted 2 lines
My apologies for review delay.
Apologies too. I was on something else.
Quoted 5 lines
As far I understand the hypothetical problem scenario is:
- CPU core reorders reading of qwords of 16B vector
- core reads the second 8B of CQE (old CQE values)
- CQE update
- core reads the first 8B of CQE (new CQE values)Yes, This is the problem.
Quoted 4 lines
How the re-reading of CQEs can resolve the issue?
This wrong scenario might happen on the second read and we would run into the same issue.Quoted 5 lines
Here we are trying to ordering reading of a 16B vector (8B with
op_own
- high, and 8B withoutop_own- low).
The first read will load 16B. The second read will load and update low 8B (noop_own).OK, I got the point, thank you for the explanations.
Can we avoid the first reading of low 8B (no containing CQE owning field)?I mean to update this part to read only upper 8Bs:
/* B.0 (CQE 3) load a block having op_own. */ c3 = vld1q_u64((uint64_t *)(p3 + 48)); /* B.0 (CQE 2) load a block having op_own. */ c2 = vld1q_u64((uint64_t *)(p2 + 48)); /* B.0 (CQE 1) load a block having op_own. */ c1 = vld1q_u64((uint64_t *)(p1 + 48)); /* B.0 (CQE 0) load a block having op_own. */ c0 = vld1q_u64((uint64_t *)(p0 + 48)); /* Synchronize for loading the rest of blocks. */ rte_io_rmb();Quoted 2 lines
Because lower 8Bs will be overlapped with the second read (in your patch) and barrier ensures the correct order.
Hi Slava,
Quoted 9 lines
Yes, your suggestion is valid.
Actually, I tried that approach: load higher 8B + barrier + load lower 8B + combine the two 8Bs into a vector.
It also has no observable performance impact but generates more instructions compared to the current patch (the 'combine' operation).
So I followed current approach.Thanks.
Quoted 1 line
Hi Slava,
Are there any further comments?
Thanks,
Ruifeng
Slava Ovsiienko
Acked-byQuoted 12 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: четверг, 29 сентября 2022 г. 09:51
To: Slava Ovsiienko <[email protected]>; Ali Alnubani <[email protected]>; Matan Azrad <[email protected]>
Cc: [email protected]; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>; nd <[email protected]>
Subject: RE: [PATCH] net/mlx5: fix risk in Rx descriptor read in NEON vector path
Hi Slava,
Quoted 2 lines
Are there any further comments?
Hi, Ruifeng
I've recalled the context and re-reviewed the patch. There is no performance impact and I'm run out of objections 😊 My sincere apologizes for the long delay ☹
PATCH v2 1/1 net/mlx5: fix risk in Rx descriptor read in NEON vector path
In reply to Ruifeng WangIn NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes op_own field could be reordered after read of other words. In this case, some words could contain invalid data.
Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
---
v2: Rebased and added tags that received.
drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +8 -0
@@ -675,6 +675,14 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, c0 =vld1q_u64 ((uint64_t *)(p0 +48 ));/* Synchronize for loading the rest of blocks. */ rte_io_rmb (); +/* B.0 (CQE 3) reload lower half of the block. */ + c3 =vld1q_lane_u64 ((uint64_t *)(p3 +48 ), c3,0 ); +/* B.0 (CQE 2) reload lower half of the block. */ + c2 =vld1q_lane_u64 ((uint64_t *)(p2 +48 ), c2,0 ); +/* B.0 (CQE 1) reload lower half of the block. */ + c1 =vld1q_lane_u64 ((uint64_t *)(p1 +48 ), c1,0 ); +/* B.0 (CQE 0) reload lower half of the block. */ + c0 =vld1q_lane_u64 ((uint64_t *)(p0 +48 ), c0,0 );/* Prefetch next 4 CQEs. */ if (pkts_n - pos >=2 *MLX5_VPMD_DESCS_PER_LOOP ) {unsigned int next = pos +MLX5_VPMD_DESCS_PER_LOOP ;
Signature · 1 line
2.25.1
Hi,
Quoted 23 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, May 30, 2023 8:48 AM
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>; Ali Alnubani <[email protected]>
Subject: [PATCH v2] net/mlx5: fix risk in Rx descriptor read in NEON vector pathIn NEON vector PMD, vector load loads two contiguous 8B of descriptor data into vector register. Given vector load ensures no 16B atomicity, read of the word that includes
op_ownfield could be reordered after read of other words. In this case, some words could contain invalid data.Reloaded qword0 after read barrier to update vector register. This ensures that the fetched data is correct.
Testpmd single core test on N1SDP/ThunderX2 showed no performance drop.
Fixes 1742c2d9fab0 ("net/mlx5: fix synchronization on polling Rx
Quoted 9 lines
completions")
Cc: [email protected]Signed-off-by Ruifeng Wang <[email protected]>Tested-by Ali Alnubani <[email protected]>---
v2: Rebased and added tags that received.
Patch applied to next-net-mlx,
Kindest regards,
Raslan Darawsheh