MLX5 PMD tuning
cover MLX5 PMD tuning
This series include optimizations for MLX5 PMD.
In tests on Arm N1SDP with MLX5 40G NIC, changes
showed performance gain.
Ruifeng Wang (2):
net/mlx5: remove redundant operations
net/mlx5: reduce unnecessary memory access
drivers/net/mlx5/mlx5_rxtx_vec.c | 6 ++++-- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 9 +-------- 2 files changed, 5 insertions(+), 10 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/2 net/mlx5: remove redundant operations
Some operations on mask are redundant and can be removed. The change yielded 1.6% performance gain on N1SDP. On ThunderX2, slight performance uplift was also observed.
--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +1 -8
@@ -768,18 +768,11 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, comp_mask), 0)) / (sizeof (uint16_t ) *8 );/* D.6 mask out entries after the compressed CQE. */ - mask =vcreate_u16 (comp_idx <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (comp_idx *sizeof (uint16_t ) *8 ) : -0 ); - invalid_mask =vorr_u16 (invalid_mask, mask); + invalid_mask =vorr_u16 (invalid_mask, comp_mask);/* D.7 count non-compressed valid CQEs. */ n =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( invalid_mask),0 )) / (sizeof (uint16_t ) *8 ); nocmp_n += n; -/* D.2 get the final invalid mask. */ - mask =vcreate_u16 (n <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (n *sizeof (uint16_t ) *8 ) :0 ); - invalid_mask =vorr_u16 (invalid_mask, mask);/* D.3 check error in opcode. */ opcode =vceq_u16 (resp_err_check, opcode); opcode =vbic_u16 (opcode, invalid_mask);
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH 1/2] net/mlx5: remove redundant operations
Hi, Ruifeng
Quoted 35 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, June 1, 2021 11:31
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>; [email protected]
Subject: [PATCH 1/2] net/mlx5: remove redundant operationsSome operations on mask are redundant and can be removed. The change yielded 1.6% performance gain on N1SDP. On ThunderX2, slight performance uplift was also observed.
Fixes 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-1@@ -768,18 +768,11 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,volatile struct mlx5_cqe *cq, comp_mask), 0)) / (sizeof(uint16_t) * 8);/* D.6 mask out entries after the compressed CQE. */ - mask = vcreate_u16(comp_idx <
MLX5_VPMD_DESCS_PER_LOOP?
drivers/net/mlx5/mlx5_rxtx_vec_neon.h rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, · line 771
Quoted 4 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +1
-3- -1UL >> (comp_idx * sizeof(uint16_t) * 8) : - 0); - invalid_mask =vorr_u16 (invalid_mask, mask); + invalid_mask =vorr_u16 (invalid_mask, comp_mask);
Mmmm... I'm not sure we can drop the masking compressed (and following) CQE skip. Let's consider the completion scenario (the series of 4 CQEs, each element is 64B long)
0: normal uncompressed CQE, ownership OK, format uncompressed, opcode OK, no error
1: compressed CQE, ownership OK, format compressed, opcode OK, no error
2: miniCQE array, format can be any!!, may be discovered as ownership OK, format uncompressed, opcode OK, no error 3: miniCQE array, format can be any!!, may be discovered as ownership OK, format uncompressed, opcode OK, no error
Obviously, we should unconditionally mask out 2 and 3, regardless of recognized their formats/opcode/error/etc. I think we can get the diff above and skip diff below:
drivers/net/mlx5/mlx5_rxtx_vec_neon.h rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, · line 782
Quoted 8 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-4/* D.7 count non-compressed valid CQEs. */ n =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( invalid_mask),0 )) / (sizeof (uint16_t ) *8 ); nocmp_n += n; -/* D.2 get the final invalid mask. */ - mask =vcreate_u16 (n <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (n *sizeof (uint16_t ) *8 ) :0 ); - invalid_mask =vorr_u16 (invalid_mask, mask);
and get the correct final invalid_mask - all compressed and invalid CQEs and following ones will be masked out.
With best regards,
Slava
Re: [dpdk-dev] [PATCH 1/2] net/mlx5: remove redundant operations
Quoted 11 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Friday, July 2, 2021 4:13 PM
To: Ruifeng Wang <[email protected]>; Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: RE: [PATCH 1/2] net/mlx5: remove redundant operationsHi, Ruifeng
Hi, Slava
Quoted 46 lines
Quoted 36 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, June 1, 2021 11:31
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>; [email protected]
Subject: [PATCH 1/2] net/mlx5: remove redundant operationsSome operations on mask are redundant and can be removed. The change yielded 1.6% performance gain on N1SDP. On ThunderX2, slight performance uplift was also observed.
Fixes 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-1@@ -768,18 +768,11 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,volatile struct mlx5_cqe *cq, comp_mask), 0)) / (sizeof(uint16_t) * 8);/* D.6 mask out entries after the compressed CQE. */ - mask = vcreate_u16(comp_idx <
MLX5_VPMD_DESCS_PER_LOOP?drivers/net/mlx5/mlx5_rxtx_vec_neon.h rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, · line 771
Quoted 4 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +1
-3- -1UL >> (comp_idx * sizeof(uint16_t) * 8) : - 0); - invalid_mask =vorr_u16 (invalid_mask, mask); + invalid_mask =vorr_u16 (invalid_mask, comp_mask);Mmmm... I'm not sure we can drop the masking compressed (and following) CQE skip.
Let's consider the completion scenario (the series of 4 CQEs, each element is 64B long)
Quoted 7 lines
0: normal uncompressed CQE, ownership OK, format uncompressed, opcode OK, no error
1: compressed CQE, ownership OK, format compressed, opcode OK, no error 2: miniCQE array, format can be any!!, may be discovered as ownership OK, format uncompressed, opcode OK, no error
3: miniCQE array, format can be any!!, may be discovered as ownership OK, format uncompressed, opcode OK, no error
Thanks for your review and explanation about CQE processing details. I did the change based on the fact that some calculations doesn't change the data. So some intermediate calculations were removed.
In the above diff section, result of 'mask' always equals to the nearest 'comp_mask' that above it. So I just remoed 'mask' and use 'comp_mask' instead.
Quoted 5 lines
Obviously, we should unconditionally mask out 2 and 3, regardless of recognized their formats/opcode/error/etc.
I think we can get the diff above and skip diff below:
Quoted 11 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, · line 782
Quoted 8 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-4/* D.7 count non-compressed valid CQEs. */ n =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( invalid_mask),0 )) / (sizeof (uint16_t ) *8 ); nocmp_n += n; -/* D.2 get the final invalid mask. */ - mask =vcreate_u16 (n <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (n *sizeof (uint16_t ) *8 ) :0 ); - invalid_mask =vorr_u16 (invalid_mask, mask);and get the correct final
invalid_mask- all compressed and invalid CQEs and following ones will be masked out.
This diff section is similar to the previous one.
'mask' always equals to the nearest 'invalid_mask' that above it. So entire line "invalid_mask = vorr_u16(invalid_mask, mask);" can be removed.
Code logic is not changed. But I'm not sure the code change impacts readability or maintainability that you may concern.
Thanks.
Quoted 3 lines
With best regards,
Slava
Re: [dpdk-dev] [PATCH 1/2] net/mlx5: remove redundant operations
Hi, Ruifeng
The invalid_mask is used to set error flags and calculate the statistics. So, all the CQEs the first one with error or invalid status should be masked out (and the CQEs after that).
IMO, what we could improve (apply just the part of the patch below):
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
index 2234fbe6b2..98a75b09c6 100644
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +1 -4
@@ -768,18 +768,11 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, comp_mask), 0)) / (sizeof (uint16_t ) *8 );/* D.6 mask out entries after the compressed CQE. */ - mask =vcreate_u16 (comp_idx <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (comp_idx *sizeof (uint16_t ) *8 ) : -0 ); - invalid_mask =vorr_u16 (invalid_mask, mask); + invalid_mask =vorr_u16 (invalid_mask, comp_mask);/* D.7 count non-compressed valid CQEs. */ n =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( invalid_mask),0 )) / (sizeof (uint16_t ) *8 ); nocmp_n += n;
<<<<
And that's it. The rest of the patch:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- /* D.2 get the final invalid mask. */
- mask = vcreate_u16(n < MLX5_VPMD_DESCS_PER_LOOP ?
- -1UL >> (n * sizeof(uint16_t) * 8) : 0);
- invalid_mask = vorr_u16(invalid_mask, mask);
<<<<
Should not be applied, otherwise the following might be affected:
opcode = vbic_u16(opcode, invalid_mask);
...
opcode = vbic_u16(opcode, invalid_mask);
With best regards,
Slava
Quoted 116 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Friday, July 2, 2021 13:30
To: Slava Ovsiienko <[email protected]>; Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 1/2] net/mlx5: remove redundant operationsQuoted 11 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Friday, July 2, 2021 4:13 PM
To: Ruifeng Wang <[email protected]>; Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: RE: [PATCH 1/2] net/mlx5: remove redundant operationsHi, Ruifeng
Hi, Slava
Quoted 9 lines
Quoted 9 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, June 1, 2021 11:31
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng WangQuoted 37 lines
Quoted 27 lines
[email protected]
Subject: [PATCH 1/2] net/mlx5: remove redundant operationsSome operations on mask are redundant and can be removed. The change yielded 1.6% performance gain on N1SDP. On ThunderX2, slight performance uplift was also observed.
Fixes 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-1@@ -768,18 +768,11 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,volatile struct mlx5_cqe *cq, comp_mask), 0)) / (sizeof(uint16_t) * 8);/* D.6 mask out entries after the compressed CQE. */ - mask = vcreate_u16(comp_idx <
MLX5_VPMD_DESCS_PER_LOOP?drivers/net/mlx5/mlx5_rxtx_vec_neon.h
Quoted 4 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +1
-3- -1UL >> (comp_idx * sizeof(uint16_t) * 8) : - 0); - invalid_mask =vorr_u16 (invalid_mask, mask); + invalid_mask =vorr_u16 (invalid_mask, comp_mask);Mmmm... I'm not sure we can drop the masking compressed (and following) CQE skip.
Let's consider the completion scenario (the series of 4 CQEs, each element is 64B long)Quoted 8 lines
0: normal uncompressed CQE, ownership OK, format uncompressed, opcode OK, no error
1: compressed CQE, ownership OK, format compressed, opcode OK, no error
2: miniCQE array, format can be any!!, may be discovered as ownership OK, format uncompressed, opcode OK, no error
3: miniCQE array, format can be any!!, may be discovered as ownership OK, format uncompressed, opcode OK, no errorThanks for your review and explanation about CQE processing details. I did the change based on the fact that some calculations doesn't change the data.
So some intermediate calculations were removed.In the above diff section, result of 'mask' always equals to the nearest '
comp_mask' that above it.
So I just remoed 'mask' and use 'comp_mask' instead.Quoted 5 lines
Obviously, we should unconditionally mask out 2 and 3, regardless of recognized their formats/opcode/error/etc.
I think we can get the diff above and skip diff below:Quoted 11 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h
Quoted 8 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-4/* D.7 count non-compressed valid CQEs. */ n =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( invalid_mask),0 )) / (sizeof (uint16_t ) *8 ); nocmp_n += n; -/* D.2 get the final invalid mask. */ - mask =vcreate_u16 (n <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (n *sizeof (uint16_t ) *8 ) :0 ); - invalid_mask =vorr_u16 (invalid_mask, mask);and get the correct final
invalid_mask- all compressed and invalid CQEs and following ones will be masked out.This diff section is similar to the previous one.
'mask' always equals to the nearest 'invalid_mask' that above it. So entire line "invalid_mask=vorr_u16(invalid_mask, mask);" can be removed.Code logic is not changed. But I'm not sure the code change impacts readability or maintainability that you may concern.
Thanks.
Quoted 3 lines
With best regards,
Slava
Re: [dpdk-dev] [PATCH 1/2] net/mlx5: remove redundant operations
Quoted 11 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Monday, July 5, 2021 6:02 PM
To: Ruifeng Wang <[email protected]>; Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 1/2] net/mlx5: remove redundant operations
Quoted 5 lines
Hi, Ruifeng
The
invalid_maskis used to set error flags and calculate the statistics. So, all the CQEs the first one with error or invalid status should be masked out (and the CQEs after that).
Now I understand it. What I was missing is inconsecutive mask bits. Thanks for your patience.
I'll update in next version.
Quoted 157 lines
IMO, what we could improve (apply just the part of the patch below):
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
index
2234fbe6b2..98a75b09c6100644drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-1@@ -768,18 +768,11 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,volatile struct mlx5_cqe *cq, comp_mask), 0)) / (sizeof(uint16_t) * 8);/* D.6 mask out entries after the compressed CQE. */ - mask = vcreate_u16(comp_idx <
MLX5_VPMD_DESCS_PER_LOOP?drivers/net/mlx5/mlx5_rxtx_vec_neon.h +1
-3- -1UL >> (comp_idx * sizeof(uint16_t) * 8) : - 0); - invalid_mask =vorr_u16 (invalid_mask, mask); + invalid_mask =vorr_u16 (invalid_mask, comp_mask);/* D.7 count non-compressed valid CQEs. */ n =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( invalid_mask),0 )) / (sizeof (uint16_t ) *8 ); nocmp_n += n;<<<<
And that's it. The rest of the patch:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-4-/* D.2 get the final invalid mask. */ - mask =vcreate_u16 (n <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (n *sizeof (uint16_t ) *8 ) :0 ); - invalid_mask =vorr_u16 (invalid_mask, mask);<<<<
Should not be applied, otherwise the following might be affected:opcode =
vbic_u16(opcode, invalid_mask); ...
opcode =vbic_u16(opcode, invalid_mask);With best regards,
SlavaQuoted 16 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Friday, July 2, 2021 13:30
To: Slava Ovsiienko <[email protected]>; Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; [email protected]; nd <[email protected]>
Subject: RE: [PATCH 1/2] net/mlx5: remove redundant operationsQuoted 5 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Friday, July 2, 2021 4:13 PM
To: Ruifeng Wang <[email protected]>; Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; ShahafShuler
Quoted 90 lines
Quoted 6 lines
<[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; [email protected]
Subject: RE: [PATCH 1/2] net/mlx5: remove redundant operationsHi, Ruifeng
Hi, Slava
Quoted 9 lines
Quoted 9 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, June 1, 2021 11:31
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng WangQuoted 38 lines
Quoted 28 lines
[email protected]
Subject: [PATCH 1/2] net/mlx5: remove redundant operationsSome operations on mask are redundant and can be removed. The change yielded 1.6% performance gain on N1SDP. On ThunderX2, slight performance uplift was also observed.
Fixes 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst forARM")
Cc: [email protected]Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-1@@ -768,18 +768,11 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq,volatile struct mlx5_cqe *cq, comp_mask), 0)) / (sizeof(uint16_t) * 8);/* D.6 mask out entries after the compressed CQE. */ - mask = vcreate_u16(comp_idx <
MLX5_VPMD_DESCS_PER_LOOP?drivers/net/mlx5/mlx5_rxtx_vec_neon.h rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, · line 771
Quoted 4 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +1
-3- -1UL >> (comp_idx * sizeof(uint16_t) * 8) : - 0); - invalid_mask =vorr_u16 (invalid_mask, mask); + invalid_mask =vorr_u16 (invalid_mask, comp_mask);Mmmm... I'm not sure we can drop the masking compressed (and following) CQE skip.
Let's consider the completion scenario (the series of 4 CQEs, each element is 64B long)Quoted 8 lines
0: normal uncompressed CQE, ownership OK, format uncompressed, opcode OK, no error
1: compressed CQE, ownership OK, format compressed, opcode OK, no error
2: miniCQE array, format can be any!!, may be discovered as ownership OK, format uncompressed, opcode OK, no error 3: miniCQE array, format can be any!!, may be discovered as ownership OK, format uncompressed, opcode OK, no errorThanks for your review and explanation about CQE processing details. I did the change based on the fact that some calculations doesn't change the data.
So some intermediate calculations were removed.In the above diff section, result of 'mask' always equals to the nearest '
comp_mask' that above it.
So I just remoed 'mask' and use 'comp_mask' instead.Quoted 5 lines
Obviously, we should unconditionally mask out 2 and 3, regardless of recognized their formats/opcode/error/etc.
I think we can get the diff above and skip diff below:Quoted 11 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq,
Quoted 8 lines
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +0
-4/* D.7 count non-compressed valid CQEs. */ n =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( invalid_mask),0 )) / (sizeof (uint16_t ) *8 ); nocmp_n += n; -/* D.2 get the final invalid mask. */ - mask =vcreate_u16 (n <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (n *sizeof (uint16_t ) *8 ) :0 ); - invalid_mask =vorr_u16 (invalid_mask, mask);and get the correct final
invalid_mask- all compressed and invalid CQEs and following ones will be masked out.Quoted 3 lines
This diff section is similar to the previous one.
'mask' always equals to the nearest 'invalid_mask' that above it. So entire line "invalid_mask=vorr_u16(invalid_mask, mask);" can beremoved.
Quoted 8 lines
Code logic is not changed. But I'm not sure the code change impacts readability or maintainability that you may concern.
Thanks.
Quoted 3 lines
With best regards,
Slava
PATCH 2/2 net/mlx5: reduce unnecessary memory access
In reply to Ruifeng WangMR btree len is a constant during Rx replenish. Moved retrieve of the value out of loop to reduce data loads. Slight performance uplift was measured on N1SDP.
--- drivers/net/mlx5/mlx5_rxtx_vec.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/net/mlx5/mlx5_rxtx_vec.c +4 -2
@@ -95,6 +95,7 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq)volatile struct mlx5_wqe_data_seg *wq = &((volatile struct mlx5_wqe_data_seg *)rxq->wqes)[elts_idx];unsigned int i; +uint16_t btree_len;if (n >= rxq->rq_repl_thresh) {MLX5_ASSERT (n >=MLX5_VPMD_RXQ_RPLNSH_THRESH (q_n));
@@ -106,6 +107,8 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq) rxq->stats.rx_nombuf += n;return ; } + + btree_len =mlx5_mr_btree_len (&rxq->mr_ctrl.cache_bh);for (i =0 ; i < n; ++i) {void *buf_addr;
@@ -119,8 +122,7 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq) wq[i].addr =rte_cpu_to_be_64 ((uintptr_t )buf_addr +RTE_PKTMBUF_HEADROOM );/* If there's a single MR, no need to replace LKey. */ -if (unlikely (mlx5_mr_btree_len (&rxq->mr_ctrl.cache_bh) - >1 )) +if (unlikely (btree_len >1 )) wq[i].lkey =mlx5_rx_mb2mr (rxq, elts[i]); } rxq->rq_ci += n;
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH 2/2] net/mlx5: reduce unnecessary memory access
Hi, Ruifeng
Could we go further and implement loop inside the conditional? Like this:
if (mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1) {
for (i = 0; i < n; ++i) {
void *buf_addr = elts[i]->buf_addr;
wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr +
RTE_PKTMBUF_HEADROOM);
wq[i].lkey = mlx5_rx_mb2mr(rxq, elts[i]);
}
} else {
for (i = 0; i < n; ++i) {
void *buf_addr = elts[i]->buf_addr;
wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr +
RTE_PKTMBUF_HEADROOM);
}
}What do you think?
Also, we should check the performance on other archs is not affected.
With best regards,
Slava
Quoted 59 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, June 1, 2021 11:31
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH 2/2] net/mlx5: reduce unnecessary memory accessMR btree len is a constant during Rx replenish. Moved retrieve of the value out of loop to reduce data loads. Slight performance uplift was measured on N1SDP.
Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)drivers/net/mlx5/mlx5_rxtx_vec.c +1
-0@@ -95,6 +95,7 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq)volatile struct mlx5_wqe_data_seg *wq = &((volatile struct mlx5_wqe_data_seg *)rxq->wqes)[elts_idx];unsigned int i; +uint16_t btree_len;if (n >= rxq->rq_repl_thresh) {MLX5_ASSERT(n >=
MLX5_VPMD_RXQ_RPLNSH_THRESH(q_n));drivers/net/mlx5/mlx5_rxtx_vec.c +3
-2@@ -106,6 +107,8 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq) rxq->stats.rx_nombuf += n;return ; } + + btree_len =mlx5_mr_btree_len (&rxq->mr_ctrl.cache_bh);for (i =0 ; i < n; ++i) {void *buf_addr;@@ -119,8 +122,7 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq) wq[i].addr =rte_cpu_to_be_64 ((uintptr_t )buf_addr +RTE_PKTMBUF_HEADROOM );/* If there's a single MR, no need to replace LKey. */ -if (unlikely (mlx5_mr_btree_len (&rxq- >mr_ctrl.cache_bh) - >1 )) +if (unlikely (btree_len >1 ))drivers/net/mlx5/mlx5_rxtx_vec.c +0
-1wq[i].lkey =mlx5_rx_mb2mr (rxq, elts[i]); } rxq->rq_ci += n; --2.25.1
Re: [dpdk-dev] [PATCH 2/2] net/mlx5: reduce unnecessary memory access
Quoted 30 lines
-----Original Message-----
From: Slava Ovsiienko <[email protected]>
Sent: Friday, July 2, 2021 3:06 PM
To: Ruifeng Wang <[email protected]>; Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>
Cc: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>
Subject: RE: [PATCH 2/2] net/mlx5: reduce unnecessary memory accessHi, Ruifeng
Could we go further and implement loop inside the conditional? Like this:
if (mlx5_mr_btree_len(&rxq->mr_ctrl.cache_bh) > 1) { for (i = 0; i < n; ++i) { void *buf_addr = elts[i]->buf_addr; wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr + RTE_PKTMBUF_HEADROOM); wq[i].lkey = mlx5_rx_mb2mr(rxq, elts[i]); } } else { for (i = 0; i < n; ++i) { void *buf_addr = elts[i]->buf_addr; wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr + RTE_PKTMBUF_HEADROOM); } }
Quoted 1 line
What do you think?
Agree. Loop inside the conditional should be more efficient.
Quoted 1 line
Also, we should check the performance on other archs is not affected.
I will also test on x86 platform that I have.
Quoted 67 lines
With best regards,
Slavadrivers/net/mlx5/mlx5_rxtx_vec.c mlx5_rx_replenish_bulk_mbuf(struct
Quoted 36 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, June 1, 2021 11:31
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH 2/2] net/mlx5: reduce unnecessary memory accessMR btree len is a constant during Rx replenish. Moved retrieve of the value out of loop to reduce data loads. Slight performance uplift was measured on N1SDP.
Signed-off-by Ruifeng Wang <[email protected]>--- drivers/net/mlx5/mlx5_rxtx_vec.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)drivers/net/mlx5/mlx5_rxtx_vec.c +1
-0@@ -95,6 +95,7 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq)volatile struct mlx5_wqe_data_seg *wq = &((volatile struct mlx5_wqe_data_seg *)rxq- wqes)[elts_idx];unsigned int i; +uint16_t btree_len;drivers/net/mlx5/mlx5_rxtx_vec.c +0
-0if (n >= rxq->rq_repl_thresh) { MLX5_ASSERT(n >=
MLX5_VPMD_RXQ_RPLNSH_THRESH(q_n));drivers/net/mlx5/mlx5_rxtx_vec.c +0
-0@@ -106,6 +107,8 @@ mlx5_rx_replenish_bulk_mbuf(struct
mlx5_rxq_datadrivers/net/mlx5/mlx5_rxtx_vec.c mlx5_rx_replenish_bulk_mbuf(struct
Quoted 10 lines
drivers/net/mlx5/mlx5_rxtx_vec.c +2
-0*rxq) rxq->stats.rx_nombuf += n;return ; } + + btree_len =mlx5_mr_btree_len (&rxq->mr_ctrl.cache_bh);for (i =0 ; i < n; ++i) {void *buf_addr;@@ -119,8 +122,7 @@ mlx5_rx_replenish_bulk_mbuf(struct
mlx5_rxq_datadrivers/net/mlx5/mlx5_rxtx_vec.c mlx5_rx_replenish_bulk_mbuf(struct · line 123
Quoted 2 lines
drivers/net/mlx5/mlx5_rxtx_vec.c +0
-0*rxq) wq[i].addr = rte_cpu_to_be_64((uintptr_t)buf_addr+
Quoted 12 lines
drivers/net/mlx5/mlx5_rxtx_vec.c +1
-2RTE_PKTMBUF_HEADROOM);/* If there's a single MR, no need to replace LKey. */ -if (unlikely (mlx5_mr_btree_len (&rxq- >mr_ctrl.cache_bh) - >1 )) +if (unlikely (btree_len >1 )) wq[i].lkey =mlx5_rx_mb2mr (rxq, elts[i]);drivers/net/mlx5/mlx5_rxtx_vec.c +0
-1} rxq->rq_ci += n; --2.25.1
Quoted 18 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Tuesday, June 1, 2021 4:31 PM
To: [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
Subject: [PATCH 0/2] MLX5 PMD tuningThis series include optimizations for MLX5 PMD.
In tests on Arm N1SDP with MLX5 40G NIC, changes showed performance gain.Ruifeng Wang (2):
net/mlx5: remove redundant operations
net/mlx5: reduce unnecessary memory access
Quoted 6 lines
drivers/net/mlx5/mlx5_rxtx_vec.c | 6 ++++-- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 9 +-------- 2 files changed, 5 insertions(+), 10 deletions(-)--
2.25.1
Ping.
Appreciate your review of these patches.
Thanks.
v2 cover MLX5 PMD tuning
In reply to Ruifeng WangThis series include optimizations for MLX5 PMD.
In tests on Arm N1SDP with MLX5 40G NIC, changes
showed performance gain.
Ruifeng Wang (2):
net/mlx5: remove redundant operations
net/mlx5: reduce unnecessary memory access
drivers/net/mlx5/mlx5_rxtx_vec.c | 35 +++++++++++++++------------ drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 11 ++++----- 2 files changed, 25 insertions(+), 21 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/2 net/mlx5: remove redundant operations
Mask of entries after the compressed CQE is covered by invalid mask of non-compressed valid CQEs. Hence remove redundant calculation on mask. The change showed slight performance uplift on N1SDP.
--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/net/mlx5/mlx5_rxtx_vec_neon.h +5 -6
@@ -767,16 +767,15 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, comp_idx =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( comp_mask),0 )) / (sizeof (uint16_t ) *8 ); -/* D.6 mask out entries after the compressed CQE. */ - mask =vcreate_u16 (comp_idx <MLX5_VPMD_DESCS_PER_LOOP ? --1UL >> (comp_idx *sizeof (uint16_t ) *8 ) : -0 ); - invalid_mask =vorr_u16 (invalid_mask, mask); + invalid_mask =vorr_u16 (invalid_mask, comp_mask);/* D.7 count non-compressed valid CQEs. */ n =__builtin_clzl (vget_lane_u64 (vreinterpret_u64_u16 ( invalid_mask),0 )) / (sizeof (uint16_t ) *8 ); nocmp_n += n; -/* D.2 get the final invalid mask. */ +/* +* D.2 mask out entries after the compressed CQE. +* get the final invalid mask. +*/ mask =vcreate_u16 (n <MLX5_VPMD_DESCS_PER_LOOP ?-1UL >> (n *sizeof (uint16_t ) *8 ) :0 ); invalid_mask =vorr_u16 (invalid_mask, mask);
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 1/2] net/mlx5: remove redundant operations
Quoted 11 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Wednesday, July 7, 2021 12:03
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>; [email protected]
Subject: [PATCH v2 1/2] net/mlx5: remove redundant operations
Quoted 8 lines
Mask of entries after the compressed CQE is covered by invalid mask of non- compressed valid CQEs. Hence remove redundant calculation on mask. The change showed slight performance uplift on N1SDP.
Fixes 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")Signed-off-by Ruifeng Wang <[email protected]>
Thank you for the patch update,
PATCH v2 2/2 net/mlx5: reduce unnecessary memory access
In reply to Ruifeng WangMR btree len is a constant during Rx replenish. Moved retrieve of the value out of loop to reduce data loads. Slight performance uplift was measured on both N1SDP and x86.
--- drivers/net/mlx5/mlx5_rxtx_vec.c | 35 ++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 15 deletions(-)
drivers/net/mlx5/mlx5_rxtx_vec.c +20 -15
@@ -106,22 +106,27 @@ mlx5_rx_replenish_bulk_mbuf(struct mlx5_rxq_data *rxq) rxq->stats.rx_nombuf += n;return ; } -for (i =0 ; i < n; ++i) { -void *buf_addr; - -/* -* In order to support the mbufs with external attached -* data buffer we should use the buf_addr pointer -* instead of rte_mbuf_buf_addr(). It touches the mbuf -* itself and may impact the performance. -*/ - buf_addr = elts[i]->buf_addr; - wq[i].addr =rte_cpu_to_be_64 ((uintptr_t )buf_addr + -RTE_PKTMBUF_HEADROOM ); -/* If there's a single MR, no need to replace LKey. */ -if (unlikely (mlx5_mr_btree_len (&rxq->mr_ctrl.cache_bh) - >1 )) +if (unlikely (mlx5_mr_btree_len (&rxq->mr_ctrl.cache_bh) >1 )) { +for (i =0 ; i < n; ++i) { +/* +* In order to support the mbufs with external attached +* data buffer we should use the buf_addr pointer +* instead of rte_mbuf_buf_addr(). It touches the mbuf +* itself and may impact the performance. +*/ +void *buf_addr = elts[i]->buf_addr; + + wq[i].addr =rte_cpu_to_be_64 ((uintptr_t )buf_addr + +RTE_PKTMBUF_HEADROOM ); wq[i].lkey =mlx5_rx_mb2mr (rxq, elts[i]); + } + }else { +for (i =0 ; i < n; ++i) { +void *buf_addr = elts[i]->buf_addr; + + wq[i].addr =rte_cpu_to_be_64 ((uintptr_t )buf_addr + +RTE_PKTMBUF_HEADROOM ); + } } rxq->rq_ci += n;/* Prevent overflowing into consumed mbufs. */
Signature · 1 line
2.25.1
Re: [dpdk-dev] [PATCH v2 2/2] net/mlx5: reduce unnecessary memory access
Quoted 10 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Wednesday, July 7, 2021 12:03
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH v2 2/2] net/mlx5: reduce unnecessary memory access
Quoted 6 lines
MR btree len is a constant during Rx replenish. Moved retrieve of the value out of loop to reduce data loads. Slight performance uplift was measured on both N1SDP and x86.
Suggested-by Slava Ovsiienko <[email protected]>Signed-off-by Ruifeng Wang <[email protected]>
Thank you for the update,
Hi,
Quoted 18 lines
-----Original Message-----
From: Ruifeng Wang <[email protected]>
Sent: Wednesday, July 7, 2021 12:03 PM
To: Raslan Darawsheh <[email protected]>; Matan Azrad <[email protected]>; Shahaf Shuler <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Ruifeng Wang <[email protected]>
Subject: [PATCH v2 0/2] MLX5 PMD tuningThis series include optimizations for MLX5 PMD.
In tests on Arm N1SDP with MLX5 40G NIC, changes showed performance gain.Ruifeng Wang (2):
net/mlx5: remove redundant operations
net/mlx5: reduce unnecessary memory access
Quoted 6 lines
drivers/net/mlx5/mlx5_rxtx_vec.c | 35 +++++++++++++++------------ drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 11 ++++----- 2 files changed, 25 insertions(+), 21 deletions(-)--
2.25.1
Series applied to next-net-mlx,
Kindest regards,
Raslan Darawsheh