Skip to contentSkip to conversation

fix Memory region lookups

PATCH Superseded Shahaf Shuler 3 participants 18 messages Jan 23, 2018 – Jan 25, 2018 Network drivers

lore
Share & tools
lore
#1 lore raw

PATCH 1/5 net/mlx5: fix Memory Region cache lookup

In reply to Shahaf Shuler

The memory registration (MR) cache contains pointers to mlx5_mr. The MR cache indexes are filled when a new MR is created. As it is possible for MR to be created on the flight, an extra validation must be added to avoid segmentation fault.

Fixes 6cb559d67b83 ("net/mlx5: add vectorized Rx/Tx burst for x86")
---
 drivers/net/mlx5/mlx5_rxtx.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx.h +2 -1
index 2eb2f0506..a63364d79 100644
@@ -555,7 +555,8 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end >= addr))
 		return txq->mp2mr[i]->lkey;
 	for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) {
-		if (unlikely(txq->mp2mr[i]->mr == NULL)) {
+		if (unlikely(txq->mp2mr[i] == NULL ||
+		    txq->mp2mr[i]->mr == NULL)) {
 			/* Unknown MP, add a new MR for it. */
 			break;
 		}
Signature · 1 line
2.12.0
#2 lore raw

PATCH 2/5 net/mlx5: fix secondary process mempool registration

In reply to Shahaf Shuler

Secondary process is not allowed to register mempools on the flight.

The code will return invalid memory key for such case.

Fixes 87ec44ce1651 ("net/mlx5: add operations for secondary process")
---
 doc/guides/nics/mlx5.rst     |  7 ++++++-
 drivers/net/mlx5/mlx5_rxtx.h | 17 ++++++++++++++---
 2 files changed, 20 insertions(+), 4 deletions(-)
doc/guides/nics/mlx5.rst +6 -1
index bdc2216c0..2f860402f 100644
@@ -106,7 +106,12 @@ Limitations
 
 - Inner RSS for VXLAN frames is not supported yet.
 - Hardware checksum RX offloads for VXLAN inner header are not supported yet.
-- Forked secondary process not supported.
+- For secondary process:
+
+  - Forked secondary process not supported.
+  - All mempools must be initialized before rte_eth_dev_start().
+  - Number of mempools must less than CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE
+
 - Flow pattern without any specific vlan will match for vlan packets as well:
 
   When VLAN spec is not specified in the pattern, the matching rule will be created with VLAN as a wild card.
drivers/net/mlx5/mlx5_rxtx.h +14 -3
index a63364d79..79cdfc793 100644
@@ -550,6 +550,7 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	uint16_t i = txq->mr_cache_idx;
 	uintptr_t addr = rte_pktmbuf_mtod(mb, uintptr_t);
 	struct mlx5_mr *mr;
+	struct rte_mempool *mp;
 
 	assert(i < RTE_DIM(txq->mp2mr));
 	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end >= addr))
@@ -563,14 +564,24 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 		if (txq->mp2mr[i]->start <= addr &&
 		    txq->mp2mr[i]->end >= addr) {
 			assert(txq->mp2mr[i]->lkey != (uint32_t)-1);
-			assert(rte_cpu_to_be_32(txq->mp2mr[i]->mr->lkey) ==
-			       txq->mp2mr[i]->lkey);
+			if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+				assert(rte_cpu_to_be_32(txq->mp2mr[i]->mr->lkey)
+				       == txq->mp2mr[i]->lkey);
+			}
 			txq->mr_cache_idx = i;
 			return txq->mp2mr[i]->lkey;
 		}
 	}
 	txq->mr_cache_idx = 0;
-	mr = mlx5_txq_mp2mr_reg(txq, mlx5_tx_mb2mp(mb), i);
+	mp = mlx5_tx_mb2mp(mb);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		WARN("Using unregistered mempool 0x%p(%s) in secondary process,"
+		     " please create mempool before rte_eth_dev_start()",
+		     (void *)mp, mp->name);
+		assert(rte_eal_process_type() == RTE_PROC_PRIMARY);
+		return (uint32_t)-1;
+	}
+	mr = mlx5_txq_mp2mr_reg(txq, mp, i);
 	/*
 	 * Request the reference to use in this queue, the original one is
 	 * kept by the control plane.
Signature · 1 line
2.12.0
#7 lore raw

PATCH 4/5 net/mlx5: fix memory registration cache last index

In reply to Shahaf Shuler

In case memory registration cache is full the new mempool will be inserted in the last index of the array.

Update the last entry being hit to reflect it.

Fixes 6cb559d67b83 ("net/mlx5: add vectorized Rx/Tx burst for x86")
---
 drivers/net/mlx5/mlx5_rxtx.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx.h +1 -1
index 2934f9fb3..4bedfb89b 100644
@@ -572,7 +572,6 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 			return txq->mp2mr[i]->lkey;
 		}
 	}
-	txq->mr_cache_idx = 0;
 	mp = mlx5_tx_mb2mp(mb);
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		WARN("Using unregistered mempool 0x%p(%s) in secondary process,"
@@ -588,6 +587,7 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	 */
 	if (mr) {
 		rte_atomic32_inc(&mr->refcnt);
+		txq->mr_cache_idx = i >= RTE_DIM(txq->mp2mr) ? i - 1 : i;
 		return mr->lkey;
 	} else {
 		WARN("Failed to register mempool 0x%p(%s)",
Signature · 1 line
2.12.0
#0 lore raw

cover fix Memory region lookups

The series contains multiple small patches which address various of issues with the Memory region lookup on the Transmit side.

Xueming Li
Shahaf Shuler (5):
net/mlx5: fix Memory Region cache lookup net/mlx5: fix secondary process mempool registration net/mlx5: assert for un-successful memory registration net/mlx5: fix memory registration cache last index net/mlx5: fix Memory Region boundary checks

 doc/guides/nics/mlx5.rst     |  7 ++++++-
 drivers/net/mlx5/mlx5_rxtx.h | 28 ++++++++++++++++++++++------
 2 files changed, 28 insertions(+), 7 deletions(-)
Signature · 1 line
2.12.0
#5 lore raw

PATCH 3/5 net/mlx5: assert for un-successful memory registration

Memory registration can fail, add the proper assert for such scenario for it at least to be visible in debug mode.

---
 drivers/net/mlx5/mlx5_rxtx.h | 4 ++++
 1 file changed, 4 insertions(+)
drivers/net/mlx5/mlx5_rxtx.h +4 -0
index 79cdfc793..2934f9fb3 100644
@@ -589,6 +589,10 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	if (mr) {
 		rte_atomic32_inc(&mr->refcnt);
 		return mr->lkey;
+	} else {
+		WARN("Failed to register mempool 0x%p(%s)",
+		      (void *)mp, mp->name);
+		assert(mr != NULL);
 	}
 	return (uint32_t)-1;
 }
Signature · 1 line
2.12.0
#8 lore raw

PATCH 5/5 net/mlx5: fix Memory Region boundary checks

In reply to Shahaf Shuler

Since commit f81ec748434b ("net/mlx5: fix memory region lookup") the memory regions (MR) are no longer overlaps.

Comparing the end address of the MR should be exclusive, otherwise two contiguous MRs may cause wrong matching.

Fixes f81ec748434b ("net/mlx5: fix memory region lookup")
---
 drivers/net/mlx5/mlx5_rxtx.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx.h +1 -1
index 4bedfb89b..692069971 100644
@@ -553,7 +553,7 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	struct rte_mempool *mp;
 
 	assert(i < RTE_DIM(txq->mp2mr));
-	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end >= addr))
+	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end > addr))
 		return txq->mp2mr[i]->lkey;
 	for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) {
 		if (unlikely(txq->mp2mr[i] == NULL ||
Signature · 1 line
2.12.0
#3 lore raw

Re: [PATCH 2/5] net/mlx5: fix secondary process mempool registration

In reply to Shahaf Shuler

Hi Shahaf,

Quoted 19 lines · Shahaf Shuler

Secondary process is not allowed to register mempools on the flight.

The code will return invalid memory key for such case.

Fixes 87ec44ce1651 ("net/mlx5: add operations for secondary process")
---
 doc/guides/nics/mlx5.rst     |  7 ++++++-
 drivers/net/mlx5/mlx5_rxtx.h | 17 ++++++++++++++---
 2 files changed, 20 insertions(+), 4 deletions(-)
doc/guides/nics/mlx5.rst +0 -0
index bdc2216c0..2f860402f 100644

doc/guides/nics/mlx5.rst Limitations · line 113

Quoted 9 lines
doc/guides/nics/mlx5.rst +5 -1
@@ -106,7 +106,12 @@ Limitations
 
 - Inner RSS for VXLAN frames is not supported yet.
 - Hardware checksum RX offloads for VXLAN inner header are not supported yet.
-- Forked secondary process not supported.
+- For secondary process:
+
+  - Forked secondary process not supported.
+  - All mempools must be initialized before rte_eth_dev_start().
+  - Number of mempools must less than CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE

Why such limitation? Registering a new memory is independent of searching a new one when the cache is too small.

Quoted 16 lines
doc/guides/nics/mlx5.rst +0 -0
 - Flow pattern without any specific vlan will match for vlan packets as well:
 
   When VLAN spec is not specified in the pattern, the matching rule will be created with VLAN as a wild card.
drivers/net/mlx5/mlx5_rxtx.h +1 -0
index a63364d79..79cdfc793 100644
@@ -550,6 +550,7 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	uint16_t i = txq->mr_cache_idx;
 	uintptr_t addr = rte_pktmbuf_mtod(mb, uintptr_t);
 	struct mlx5_mr *mr;
+	struct rte_mempool *mp;
 
 	assert(i < RTE_DIM(txq->mp2mr));
 	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end >= addr))

drivers/net/mlx5/mlx5_rxtx.h mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb) · line 570

Quoted 9 lines
drivers/net/mlx5/mlx5_rxtx.h +4 -2
@@ -563,14 +564,24 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 		if (txq->mp2mr[i]->start <= addr &&
 		    txq->mp2mr[i]->end >= addr) {
 			assert(txq->mp2mr[i]->lkey != (uint32_t)-1);
-			assert(rte_cpu_to_be_32(txq->mp2mr[i]->mr->lkey) ==
-			       txq->mp2mr[i]->lkey);
+			if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+				assert(rte_cpu_to_be_32(txq->mp2mr[i]->mr->lkey)
+				       == txq->mp2mr[i]->lkey);
+			}

This code should be inside priv_txq_mp2mr_reg() to let the secondary search inside the MR list when the cache is too small.

If it does not find any MR it should fail before calling priv_mr_new().

Quoted 20 lines
drivers/net/mlx5/mlx5_rxtx.h +9 -1
 			txq->mr_cache_idx = i;
 			return txq->mp2mr[i]->lkey;
 		}
 	}
 	txq->mr_cache_idx = 0;
-	mr = mlx5_txq_mp2mr_reg(txq, mlx5_tx_mb2mp(mb), i);
+	mp = mlx5_tx_mb2mp(mb);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		WARN("Using unregistered mempool 0x%p(%s) in secondary process,"
+		     " please create mempool before rte_eth_dev_start()",
+		     (void *)mp, mp->name);
+		assert(rte_eal_process_type() == RTE_PROC_PRIMARY);
+		return (uint32_t)-1;
+	}
+	mr = mlx5_txq_mp2mr_reg(txq, mp, i);
 	/*
 	 * Request the reference to use in this queue, the original one is
 	 * kept by the control plane.
Signature · 1 line
2.12.0
Signature · 2 lines
Nélio Laranjeiro
6WIND
#6 lore raw

Re: [PATCH 3/5] net/mlx5: assert for un-successful memory registration

In reply to Shahaf Shuler

Hi Shahaf,

Quoted 14 lines · Shahaf Shuler

Memory registration can fail, add the proper assert for such scenario for it at least to be visible in debug mode.

---
 drivers/net/mlx5/mlx5_rxtx.h | 4 ++++
 1 file changed, 4 insertions(+)
drivers/net/mlx5/mlx5_rxtx.h +0 -0
index 79cdfc793..2934f9fb3 100644

drivers/net/mlx5/mlx5_rxtx.h mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb) · line 595

Quoted 7 lines
drivers/net/mlx5/mlx5_rxtx.h +4 -0
@@ -589,6 +589,10 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	if (mr) {
 		rte_atomic32_inc(&mr->refcnt);
 		return mr->lkey;
+	} else {
+		WARN("Failed to register mempool 0x%p(%s)",
+		      (void *)mp, mp->name);
+		assert(mr != NULL);

This assert seems wrong.

Why this assert, you don't trust the CPU to verify the pointer is NULL?

Quoted 5 lines
drivers/net/mlx5/mlx5_rxtx.h +0 -0
 	}
 	return (uint32_t)-1;
 }
Signature · 1 line
2.12.0

Regards,

Signature · 2 lines
Nélio Laranjeiro
6WIND
#4 lore raw

Re: [PATCH 2/5] net/mlx5: fix secondary process mempool registration

In reply to Nélio Laranjeiro

Wednesday, January 24, 2018 10:14 AM, Nélio Laranjeiro:

Quoted 61 lines

Hi Shahaf,

Quoted 19 lines · Shahaf Shuler

Secondary process is not allowed to register mempools on the flight.

The code will return invalid memory key for such case.

Fixes 87ec44ce1651 ("net/mlx5: add operations for secondary process")
---
 doc/guides/nics/mlx5.rst     |  7 ++++++-
 drivers/net/mlx5/mlx5_rxtx.h | 17 ++++++++++++++---
 2 files changed, 20 insertions(+), 4 deletions(-)
doc/guides/nics/mlx5.rst index renamed from doc/guides/nics/mlx5.rst +0 -0

bdc2216c0..2f860402f 100644

doc/guides/nics/mlx5.rst Limitations · line 108

Quoted 3 lines
doc/guides/nics/mlx5.rst +0 -0
@@ -106,7 +106,12 @@ Limitations
 
 - Inner RSS for VXLAN frames is not supported yet.
 - Hardware checksum RX offloads for VXLAN inner header are not

supported yet.

doc/guides/nics/mlx5.rst Limitations

Quoted 7 lines
doc/guides/nics/mlx5.rst +6 -1
-- Forked secondary process not supported.
+- For secondary process:
+
+  - Forked secondary process not supported.
+  - All mempools must be initialized before rte_eth_dev_start().
+  - Number of mempools must less than
+ CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE

Why such limitation? Registering a new memory is independent of searching a new one when the cache is too small.

doc/guides/nics/mlx5.rst Limitations

Quoted 3 lines
doc/guides/nics/mlx5.rst +0 -0
 - Flow pattern without any specific vlan will match for vlan packets as well:
 
   When VLAN spec is not specified in the pattern, the matching rule will be

created with VLAN as a wild card.

drivers/net/mlx5/mlx5_rxtx.h mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct

Quoted 5 lines
(quoted diff) +0 -0

b/drivers/net/mlx5/mlx5_rxtx.h index a63364d79..79cdfc793 100644

drivers/net/mlx5/mlx5_rxtx.h +0 -0
@@ -550,6 +550,7 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct

rte_mbuf *mb)

Quoted 8 lines
drivers/net/mlx5/mlx5_rxtx.h +1 -0
 	uint16_t i = txq->mr_cache_idx;
 	uintptr_t addr = rte_pktmbuf_mtod(mb, uintptr_t);
 	struct mlx5_mr *mr;
+	struct rte_mempool *mp;
 
 	assert(i < RTE_DIM(txq->mp2mr));
 	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end >=

addr)) @@ -563,14 +564,24 @@ mlx5_tx_mb2mr(struct mlx5_txq_data

*txq, struct rte_mbuf *mb)

drivers/net/mlx5/mlx5_rxtx.h mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)

Quoted 4 lines
drivers/net/mlx5/mlx5_rxtx.h +0 -1
 		if (txq->mp2mr[i]->start <= addr &&
 		    txq->mp2mr[i]->end >= addr) {
 			assert(txq->mp2mr[i]->lkey != (uint32_t)-1);
-			assert(rte_cpu_to_be_32(txq->mp2mr[i]->mr->lkey)

==

Quoted 9 lines

drivers/net/mlx5/mlx5_rxtx.h

Quoted 6 lines
drivers/net/mlx5/mlx5_rxtx.h +2 -1
-			       txq->mp2mr[i]->lkey);
+			if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+				assert(rte_cpu_to_be_32(txq->mp2mr[i]-

mr->lkey)

drivers/net/mlx5/mlx5_rxtx.h +2 -0
+				       == txq->mp2mr[i]->lkey);
+			}

This code should be inside priv_txq_mp2mr_reg() to let the secondary search inside the MR list when the cache is too small.

You probably mean below code and not above. Yes good suggestion.
Regarding the above code - the assert in un-accessible from the secondary process as it tries to access ibv_mr. I will add another patch to remove this assert completely (in order to save the comparison logic).

Quoted 28 lines

If it does not find any MR it should fail before calling priv_mr_new().

drivers/net/mlx5/mlx5_rxtx.h mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)

Quoted 9 lines
drivers/net/mlx5/mlx5_rxtx.h +3 -1
 			txq->mr_cache_idx = i;
 			return txq->mp2mr[i]->lkey;
 		}
 	}
 	txq->mr_cache_idx = 0;
-	mr = mlx5_txq_mp2mr_reg(txq, mlx5_tx_mb2mp(mb), i);
+	mp = mlx5_tx_mb2mp(mb);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		WARN("Using unregistered mempool 0x%p(%s) in secondary

process,"

Quoted 11 lines
drivers/net/mlx5/mlx5_rxtx.h +6 -1
+		     " please create mempool before rte_eth_dev_start()",
+		     (void *)mp, mp->name);
+		assert(rte_eal_process_type() == RTE_PROC_PRIMARY);
+		return (uint32_t)-1;
+	}
+	mr = mlx5_txq_mp2mr_reg(txq, mp, i);
 	/*
 	 * Request the reference to use in this queue, the original one is
 	 * kept by the control plane.
--

2.12.0

--
Nélio Laranjeiro
6WIND

#10 lore raw

PATCH v2 1/6 net/mlx5: fix Memory Region cache lookup

In reply to Shahaf Shuler

The Memory Region (MR) cache contains pointers to mlx5_mr. The MR cache indexes are filled when a new MR is created. As it is possible for MR to be created on the flight, an extra validation must be added to avoid segmentation fault.

Fixes b0b093845793 ("net/mlx5: use buffer address for LKEY search")
---
 drivers/net/mlx5/mlx5_rxtx.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx.h +2 -1
index 2eb2f0506..a63364d79 100644
@@ -555,7 +555,8 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end >= addr))
 		return txq->mp2mr[i]->lkey;
 	for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) {
-		if (unlikely(txq->mp2mr[i]->mr == NULL)) {
+		if (unlikely(txq->mp2mr[i] == NULL ||
+		    txq->mp2mr[i]->mr == NULL)) {
 			/* Unknown MP, add a new MR for it. */
 			break;
 		}
Signature · 1 line
2.12.0
#11 lore raw

PATCH v2 2/6 net/mlx5: fix secondary process mempool registration

In reply to Shahaf Shuler

Secondary process is not allowed to register mempools on the flight.

The code will return invalid memory key for such case.

Fixes 87ec44ce1651 ("net/mlx5: add operations for secondary process")
---
 doc/guides/nics/mlx5.rst   |  6 +++++-
 drivers/net/mlx5/mlx5_mr.c | 12 ++++++++++--
 2 files changed, 15 insertions(+), 3 deletions(-)
doc/guides/nics/mlx5.rst +5 -1
index bdc2216c0..2e6d1e45a 100644
@@ -106,7 +106,11 @@ Limitations
 
 - Inner RSS for VXLAN frames is not supported yet.
 - Hardware checksum RX offloads for VXLAN inner header are not supported yet.
-- Forked secondary process not supported.
+- For secondary process:
+
+  - Forked secondary process not supported.
+  - All mempools must be initialized before rte_eth_dev_start().
+
 - Flow pattern without any specific vlan will match for vlan packets as well:
 
   When VLAN spec is not specified in the pattern, the matching rule will be created with VLAN as a wild card.
drivers/net/mlx5/mlx5_mr.c +10 -2
index 2776dc700..cb625dc61 100644
@@ -141,8 +141,16 @@ priv_txq_mp2mr_reg(struct priv *priv, struct mlx5_txq_data *txq,
 	DEBUG("%p: discovered new memory pool \"%s\" (%p)",
 	      (void *)txq_ctrl, mp->name, (void *)mp);
 	mr = priv_mr_get(priv, mp);
-	if (mr == NULL)
-		mr = priv_mr_new(priv, mp);
+	if (mr == NULL) {
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			DEBUG("Using unregistered mempool 0x%p(%s) in secondary process,"
+			     " please create mempool before rte_eth_dev_start()",
+			     (void *)mp, mp->name);
+			return NULL;
+		} else {
+			mr = priv_mr_new(priv, mp);
+		}
+	}
 	if (unlikely(mr == NULL)) {
 		DEBUG("%p: unable to configure MR, ibv_reg_mr() failed.",
 		      (void *)txq_ctrl);
Signature · 1 line
2.12.0
#14 lore raw

PATCH v2 5/6 net/mlx5: fix Memory Region cache last index

In reply to Shahaf Shuler

In case Memory Region cache is full, the new mempool will be inserted in the last index of the array.

Update the last entry being hit to reflect it.

Fixes b0b093845793 ("net/mlx5: use buffer address for LKEY search")
---
 drivers/net/mlx5/mlx5_rxtx.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/mlx5/mlx5_rxtx.h +1 -1
index e6d163a59..7e7db0e8c 100644
@@ -567,7 +567,6 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 			return txq->mp2mr[i]->lkey;
 		}
 	}
-	txq->mr_cache_idx = 0;
 	mr = mlx5_txq_mp2mr_reg(txq, mlx5_tx_mb2mp(mb), i);
 	/*
 	 * Request the reference to use in this queue, the original one is
@@ -575,6 +574,7 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	 */
 	if (mr) {
 		rte_atomic32_inc(&mr->refcnt);
+		txq->mr_cache_idx = i >= RTE_DIM(txq->mp2mr) ? i - 1 : i;
 		return mr->lkey;
 	} else {
 		struct rte_mempool *mp = mlx5_tx_mb2mp(mb);
Signature · 1 line
2.12.0
#9 lore raw

v2 cover fix Memory region lookups

In reply to Shahaf Shuler

The series contains multiple small patches which address various of issues with the Memory region lookup on the Transmit side.

On v2:
- Addressed Nelio's comments.
- Extended the "fix Memory Region boundary checks" commit to cover identical cases.
- Fixed the commits log.

Xueming Li
Shahaf Shuler (6):
net/mlx5: fix Memory Region cache lookup
net/mlx5: fix secondary process mempool registration net/mlx5: remove assert un-accessible from secondary process net/mlx5: warn for un-successful memory registration net/mlx5: fix Memory Region cache last index net/mlx5: fix Memory Region boundary checks

 doc/guides/nics/mlx5.rst     |  6 +++++-
 drivers/net/mlx5/mlx5_mr.c   | 12 ++++++++++--
 drivers/net/mlx5/mlx5_rxtx.h | 16 ++++++++++------
 3 files changed, 25 insertions(+), 9 deletions(-)
Signature · 1 line
2.12.0
#12 lore raw

PATCH v2 3/6 net/mlx5: remove assert un-accessible from secondary process

Verbs structs such as ibv_mr are not accessible from the secondary process.

Choose to remove the assert in favor of performing more checks on the critical data path.

---
 drivers/net/mlx5/mlx5_rxtx.h | 2 --
 1 file changed, 2 deletions(-)
drivers/net/mlx5/mlx5_rxtx.h +0 -2
index a63364d79..26be206a7 100644
@@ -563,8 +563,6 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 		if (txq->mp2mr[i]->start <= addr &&
 		    txq->mp2mr[i]->end >= addr) {
 			assert(txq->mp2mr[i]->lkey != (uint32_t)-1);
-			assert(rte_cpu_to_be_32(txq->mp2mr[i]->mr->lkey) ==
-			       txq->mp2mr[i]->lkey);
 			txq->mr_cache_idx = i;
 			return txq->mp2mr[i]->lkey;
 		}
Signature · 1 line
2.12.0
#13 lore raw

PATCH v2 4/6 net/mlx5: warn for un-successful memory registration

In reply to Shahaf Shuler

Memory registration can fail, add the proper warning for such scenario for it at least to be visible in debug mode.

---
 drivers/net/mlx5/mlx5_rxtx.h | 5 +++++
 1 file changed, 5 insertions(+)
drivers/net/mlx5/mlx5_rxtx.h +5 -0
index 26be206a7..e6d163a59 100644
@@ -576,6 +576,11 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	if (mr) {
 		rte_atomic32_inc(&mr->refcnt);
 		return mr->lkey;
+	} else {
+		struct rte_mempool *mp = mlx5_tx_mb2mp(mb);
+
+		WARN("Failed to register mempool 0x%p(%s)",
+		      (void *)mp, mp->name);
 	}
 	return (uint32_t)-1;
 }
Signature · 1 line
2.12.0
#15 lore raw

PATCH v2 6/6 net/mlx5: fix Memory Region boundary checks

In reply to Shahaf Shuler

Since commit f81ec748434b ("net/mlx5: fix memory region lookup") the Memory Region (MR) are no longer ovetrlaps.

Comparing the end address of the MR should be exclusive, otherwise two contiguous MRs may cause wrong matching.

Fixes f81ec748434b ("net/mlx5: fix memory region lookup")
---
 drivers/net/mlx5/mlx5_rxtx.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/mlx5/mlx5_rxtx.h +2 -2
index 7e7db0e8c..2919a747c 100644
@@ -552,7 +552,7 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 	struct mlx5_mr *mr;
 
 	assert(i < RTE_DIM(txq->mp2mr));
-	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end >= addr))
+	if (likely(txq->mp2mr[i]->start <= addr && txq->mp2mr[i]->end > addr))
 		return txq->mp2mr[i]->lkey;
 	for (i = 0; (i != RTE_DIM(txq->mp2mr)); ++i) {
 		if (unlikely(txq->mp2mr[i] == NULL ||
@@ -561,7 +561,7 @@ mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb)
 			break;
 		}
 		if (txq->mp2mr[i]->start <= addr &&
-		    txq->mp2mr[i]->end >= addr) {
+		    txq->mp2mr[i]->end > addr) {
 			assert(txq->mp2mr[i]->lkey != (uint32_t)-1);
 			txq->mr_cache_idx = i;
 			return txq->mp2mr[i]->lkey;
Signature · 1 line
2.12.0
#17 lore raw
In reply to Shahaf Shuler

Thursday, January 25, 2018 6:18 PM, Shahaf Shuler:

Quoted 20 lines

Subject: [dpdk-dev] [PATCH v2 0/6] fix Memory region lookups

The series contains multiple small patches which address various of issues with the Memory region lookup on the Transmit side.

On v2:
- Addressed Nelio's comments.
- Extended the "fix Memory Region boundary checks" commit to cover identical cases.
- Fixed the commits log.

Xueming Li
Shahaf Shuler (6):
net/mlx5: fix Memory Region cache lookup
net/mlx5: fix secondary process mempool registration net/mlx5: remove assert un-accessible from secondary process net/mlx5: warn for un-successful memory registration net/mlx5: fix Memory Region cache last index net/mlx5: fix Memory Region boundary checks

Quoted 4 lines
 doc/guides/nics/mlx5.rst     |  6 +++++-
 drivers/net/mlx5/mlx5_mr.c   | 12 ++++++++++--
 drivers/net/mlx5/mlx5_rxtx.h | 16 ++++++++++------
 3 files changed, 25 insertions(+), 9 deletions(-)

Series applied to next-net-mlx, thanks.

Quoted 3 lines

--
2.12.0

Acked-by on v2 6/6
#16 lore raw

Re: [PATCH v2 6/6] net/mlx5: fix Memory Region boundary checks

In reply to Shahaf Shuler
Quoted 9 lines · Shahaf Shuler

Since commit f81ec748434b ("net/mlx5: fix memory region lookup") the Memory Region (MR) are no longer ovetrlaps.

Comparing the end address of the MR should be exclusive, otherwise two contiguous MRs may cause wrong matching.

Quoted 7 lines
Fixes f81ec748434b ("net/mlx5: fix memory region lookup")

---

For the series,

Thanks