Skip to contentSkip to conversation

net/cxgbe: release port resources during port close

PATCH In review Rahul Lakkireddy 2 participants 7 messages Sep 1, 2020 – Sep 18, 2020 Network drivers

lore
Share & tools
lore
#0 lore raw

cover net/cxgbe: release port resources during port close

Patch 1 fixes queue DMA ring leaks seen during port close.

Patch 2 enables RTE_ETH_DEV_CLOSE_REMOVE and reworks resources release logic for port close.

Thanks,
Rahul

Rahul Lakkireddy (2):
net/cxgbe: fix queue DMA ring leaks during port close net/cxgbe: release port resources during port close

 drivers/net/cxgbe/base/adapter.h   |   1 +
 drivers/net/cxgbe/cxgbe_ethdev.c   |  31 +++++--
 drivers/net/cxgbe/cxgbe_main.c     |  12 +--
 drivers/net/cxgbe/cxgbevf_ethdev.c |   8 +-
 drivers/net/cxgbe/cxgbevf_main.c   |   2 +
 drivers/net/cxgbe/sge.c            | 144 ++++++++++++++---------------
 6 files changed, 101 insertions(+), 97 deletions(-)
Signature · 1 line
2.24.0
#1 lore raw

PATCH 1/2 net/cxgbe: fix queue DMA ring leaks during port close

Free up the DMA memzones properly for all the port's queues during port close. So, rework DMA ring allocation/free logic to use rte_eth_dma_zone_reserve()/rte_eth_dma_zone_free() helper functions for allocating/freeing the memzones.

The firmware event queue doesn't have an associated freelist queue. So, remove check that tries to give memzone name for a non-existent freelist queue.

Also, add a missing free for the control queue mempools.

Fixes 0462d115441d ("cxgbe: add device related operations")
---
 drivers/net/cxgbe/base/adapter.h |   1 +
 drivers/net/cxgbe/cxgbe_ethdev.c |   7 +-
 drivers/net/cxgbe/sge.c          | 144 +++++++++++++++----------------
 3 files changed, 70 insertions(+), 82 deletions(-)
drivers/net/cxgbe/base/adapter.h +1 -0
index 62de35c7c..3e74177f2 100644
@@ -825,6 +825,7 @@ int t4_sge_eth_rxq_start(struct adapter *adap, struct sge_rspq *rq);
 int t4_sge_eth_rxq_stop(struct adapter *adap, struct sge_rspq *rq);
 void t4_sge_eth_rxq_release(struct adapter *adap, struct sge_eth_rxq *rxq);
 void t4_sge_eth_clear_queues(struct port_info *pi);
+void t4_sge_eth_release_queues(struct port_info *pi);
 int cxgb4_set_rspq_intr_params(struct sge_rspq *q, unsigned int us,
 			       unsigned int cnt);
 int cxgbe_poll(struct sge_rspq *q, struct rte_mbuf **rx_pkts,
drivers/net/cxgbe/cxgbe_ethdev.c +1 -6
index 7c6016d5c..913af2df7 100644
@@ -326,12 +326,7 @@ void cxgbe_dev_close(struct rte_eth_dev *eth_dev)
 		return;
 
 	cxgbe_down(pi);
-
-	/*
-	 *  We clear queues only if both tx and rx path of the port
-	 *  have been disabled
-	 */
-	t4_sge_eth_clear_queues(pi);
+	t4_sge_eth_release_queues(pi);
 }
 
 /* Start the device.
drivers/net/cxgbe/sge.c +68 -76

248 lines not on this page · raw message

Signature · 1 line
2.24.0
#2 lore raw

PATCH 2/2 net/cxgbe: release port resources during port close

In reply to Rahul Lakkireddy

Enable RTE_ETH_DEV_CLOSE_REMOVE during PCI probe for all ports enumerated under the PF. Free up the underlying port Virtual Identifier (VI) and associated resources during port close. Once all the ports under the PF are closed, free up the PF-wide shared resources. Invoke port close function of all ports under the PF, in PCI remove too.

---
 drivers/net/cxgbe/cxgbe_ethdev.c   | 28 ++++++++++++++++++++++++----
 drivers/net/cxgbe/cxgbe_main.c     | 12 ++----------
 drivers/net/cxgbe/cxgbevf_ethdev.c |  8 +++++---
 drivers/net/cxgbe/cxgbevf_main.c   |  2 ++
 4 files changed, 33 insertions(+), 17 deletions(-)
drivers/net/cxgbe/cxgbe_ethdev.c +24 -4
index 913af2df7..60d325723 100644
@@ -317,16 +317,34 @@ int cxgbe_dev_mtu_set(struct rte_eth_dev *eth_dev, uint16_t mtu)
  */
 void cxgbe_dev_close(struct rte_eth_dev *eth_dev)
 {
-	struct port_info *pi = eth_dev->data->dev_private;
+	struct port_info *temp_pi, *pi = eth_dev->data->dev_private;
 	struct adapter *adapter = pi->adapter;
+	u8 i;
 
 	CXGBE_FUNC_TRACE();
 
 	if (!(adapter->flags & FULL_INIT_DONE))
 		return;
 
+	if (!pi->viid)
+		return;
+
 	cxgbe_down(pi);
 	t4_sge_eth_release_queues(pi);
+	t4_free_vi(adapter, adapter->mbox, adapter->pf, 0, pi->viid);
+	pi->viid = 0;
+
+	/* Free up the adapter-wide resources only after all the ports
+	 * under this PF have been closed.
+	 */
+	for_each_port(adapter, i) {
+		temp_pi = adap2pinfo(adapter, i);
+		if (temp_pi->viid)
+			return;
+	}
+
+	cxgbe_close(adapter);
+	rte_free(adapter);
 }
 
 /* Start the device.
@@ -1204,11 +1222,13 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
 
 static int eth_cxgbe_dev_uninit(struct rte_eth_dev *eth_dev)
 {
-	struct port_info *pi = eth_dev->data->dev_private;
-	struct adapter *adap = pi->adapter;
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+	uint16_t port_id;
 
 	/* Free up other ports and all resources */
-	cxgbe_close(adap);
+	RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device)
+		rte_eth_dev_close(port_id);
+
 	return 0;
 }
 
drivers/net/cxgbe/cxgbe_main.c +2 -10
index 2656369c5..d0a64229c 100644
@@ -1975,9 +1975,6 @@ int cxgbe_down(struct port_info *pi)
  */
 void cxgbe_close(struct adapter *adapter)
 {
-	struct port_info *pi;
-	int i;
-
 	if (adapter->flags & FULL_INIT_DONE) {
 		tid_free(&adapter->tids);
 		t4_cleanup_mpstcam(adapter);
@@ -1988,13 +1985,6 @@ void cxgbe_close(struct adapter *adapter)
 			t4_intr_disable(adapter);
 		t4_sge_tx_monitor_stop(adapter);
 		t4_free_sge_resources(adapter);
-		for_each_port(adapter, i) {
-			pi = adap2pinfo(adapter, i);
-			if (pi->viid != 0)
-				t4_free_vi(adapter, adapter->mbox,
-					   adapter->pf, 0, pi->viid);
-			rte_eth_dev_release_port(pi->eth_dev);
-		}
 		adapter->flags &= ~FULL_INIT_DONE;
 	}
 
@@ -2156,6 +2146,8 @@ int cxgbe_probe(struct adapter *adapter)
 			goto out_free;
 		}
 
+		pi->eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
 		if (i > 0) {
 			/* First port will be notified by upper layer */
 			rte_eth_dev_probing_finish(eth_dev);
drivers/net/cxgbe/cxgbevf_ethdev.c +5 -3
index 4165ba986..b3c885d6e 100644
@@ -181,11 +181,13 @@ static int eth_cxgbevf_dev_init(struct rte_eth_dev *eth_dev)
 
 static int eth_cxgbevf_dev_uninit(struct rte_eth_dev *eth_dev)
 {
-	struct port_info *pi = eth_dev->data->dev_private;
-	struct adapter *adap = pi->adapter;
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+	uint16_t port_id;
 
 	/* Free up other ports and all resources */
-	cxgbe_close(adap);
+	RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device)
+		rte_eth_dev_close(port_id);
+
 	return 0;
 }
 
drivers/net/cxgbe/cxgbevf_main.c +2 -0
index 66fb92375..9fe0ec6f6 100644
@@ -261,6 +261,8 @@ int cxgbevf_probe(struct adapter *adapter)
 			goto out_free;
 		}
 
+		pi->eth_dev->data->dev_flags |= RTE_ETH_DEV_CLOSE_REMOVE;
+
 		if (i > 0) {
 			/* First port will be notified by upper layer */
 			rte_eth_dev_probing_finish(eth_dev);
Signature · 1 line
2.24.0
#3 lore raw
Quoted 8 lines · Rahul Lakkireddy

Enable RTE_ETH_DEV_CLOSE_REMOVE during PCI probe for all ports enumerated under the PF. Free up the underlying port Virtual Identifier (VI) and associated resources during port close. Once all the ports under the PF are closed, free up the PF-wide shared resources. Invoke port close function of all ports under the PF, in PCI remove too.

<...>

drivers/net/cxgbe/cxgbe_ethdev.c static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev) · line 1230

Quoted 13 lines
drivers/net/cxgbe/cxgbe_ethdev.c +4 -3
@@ -1204,11 +1222,13 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
 
  static int eth_cxgbe_dev_uninit(struct rte_eth_dev *eth_dev)
  {
-	struct port_info *pi = eth_dev->data->dev_private;
-	struct adapter *adap = pi->adapter;
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+	uint16_t port_id;
 
  	/* Free up other ports and all resources */
-	cxgbe_close(adap);
+	RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device)
+		rte_eth_dev_close(port_id);

Is there a reason to call the ethdev wrapper API here? Why not calling 'cxgbe_dev_close()' directly?

#4 lore raw
Quoted 12 lines · Ferruh Yigit
Quoted 8 lines · Rahul Lakkireddy

Enable RTE_ETH_DEV_CLOSE_REMOVE during PCI probe for all ports enumerated under the PF. Free up the underlying port Virtual Identifier (VI) and associated resources during port close. Once all the ports under the PF are closed, free up the PF-wide shared resources. Invoke port close function of all ports under the PF, in PCI remove too.

<...>

Quoted 14 lines

drivers/net/cxgbe/cxgbe_ethdev.c static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev) · line 1228

Quoted 11 lines
drivers/net/cxgbe/cxgbe_ethdev.c +4 -3
@@ -1204,11 +1222,13 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
 static int eth_cxgbe_dev_uninit(struct rte_eth_dev *eth_dev)
 {
-	struct port_info *pi = eth_dev->data->dev_private;
-	struct adapter *adap = pi->adapter;
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+	uint16_t port_id;
 	/* Free up other ports and all resources */
-	cxgbe_close(adap);
+	RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device)
+		rte_eth_dev_close(port_id);

Is there a reason to call the ethdev wrapper API here? Why not calling 'cxgbe_dev_close()' directly?

Chelsio NICs enumerate all physical ports under a single Physical Function 4 (PF4). When PCI remove is invoked without rte_eth_dev_close() first, the rte_eth_dev_release_port() is only invoked for the first port instantiated on PF4. So, to ensure rte_eth_dev_release_port() is invoked for the remaining ports, we're explicitly calling rte_eth_dev_close() for all ports here.

Here's a testpmd example that leads to the problem.

testpmd> port stop all
Stopping ports...
Checking link statuses...
Done
testpmd> device detach 0000:03:00.4
Removing a device...
Port 0 is now closed
Port 1 is now closed
Device 0000:03:00.4 is detached
Now total ports is 1
Done
testpmd>

The Chelsio NIC has 2 ports enumerated under 0000:03:00.4. When PCI remove is invoked, rte_eth_dev_release_port() is only invoked once for the first port. You can see that the other port is still active, as printed by "Now total ports is 1".

The sequence of calls is as follows from driver's PCI remove:

eth_cxgbe_pci_remove()
rte_eth_dev_pci_generic_remove()
rte_eth_dev_pci_release()
rte_eth_dev_release_port() <--- Only invoked once

If I explicitly invoke rte_eth_dev_close() in PCI remove for all ports, as done in this patch, rte_eth_dev_release_port() is invoked for both the ports.

testpmd> port stop all
Stopping ports...
Checking link statuses...
Done
testpmd> device detach 0000:03:00.4
Removing a device...
Port 0 is now closed
Port 1 is now closed
Device 0000:03:00.4 is detached
Now total ports is 0
Done

Now, both ports are released properly as printed by "Now total ports is 0".

Thanks,
Rahul

#5 lore raw
Quoted 79 lines · Rahul Lakkireddy
Quoted 12 lines · Ferruh Yigit
Quoted 8 lines · Rahul Lakkireddy

Enable RTE_ETH_DEV_CLOSE_REMOVE during PCI probe for all ports enumerated under the PF. Free up the underlying port Virtual Identifier (VI) and associated resources during port close. Once all the ports under the PF are closed, free up the PF-wide shared resources. Invoke port close function of all ports under the PF, in PCI remove too.

<...>

Quoted 14 lines

drivers/net/cxgbe/cxgbe_ethdev.c static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev) · line 1228

Quoted 11 lines
drivers/net/cxgbe/cxgbe_ethdev.c +4 -3
@@ -1204,11 +1222,13 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
  static int eth_cxgbe_dev_uninit(struct rte_eth_dev *eth_dev)
  {
-	struct port_info *pi = eth_dev->data->dev_private;
-	struct adapter *adap = pi->adapter;
+	struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+	uint16_t port_id;
  	/* Free up other ports and all resources */
-	cxgbe_close(adap);
+	RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device)
+		rte_eth_dev_close(port_id);

Is there a reason to call the ethdev wrapper API here? Why not calling 'cxgbe_dev_close()' directly?

Chelsio NICs enumerate all physical ports under a single Physical Function 4 (PF4). When PCI remove is invoked without rte_eth_dev_close() first, the rte_eth_dev_release_port() is only invoked for the first port instantiated on PF4. So, to ensure rte_eth_dev_release_port() is invoked for the remaining ports, we're explicitly calling rte_eth_dev_close() for all ports here.

Here's a testpmd example that leads to the problem.

testpmd> port stop all
Stopping ports...
Checking link statuses...
Done
testpmd> device detach 0000:03:00.4
Removing a device...
Port 0 is now closed
Port 1 is now closed
Device 0000:03:00.4 is detached
Now total ports is 1
Done
testpmd>

The Chelsio NIC has 2 ports enumerated under 0000:03:00.4. When PCI remove is invoked, rte_eth_dev_release_port() is only invoked once for the first port. You can see that the other port is still active, as printed by "Now total ports is 1".

The sequence of calls is as follows from driver's PCI remove:

eth_cxgbe_pci_remove()
rte_eth_dev_pci_generic_remove()
rte_eth_dev_pci_release()
rte_eth_dev_release_port() <--- Only invoked once

If I explicitly invoke rte_eth_dev_close() in PCI remove for all ports, as done in this patch, rte_eth_dev_release_port() is invoked for both the ports.

testpmd> port stop all
Stopping ports...
Checking link statuses...
Done
testpmd> device detach 0000:03:00.4
Removing a device...
Port 0 is now closed
Port 1 is now closed
Device 0000:03:00.4 is detached
Now total ports is 0
Done

Quoted 3 lines

Now, both ports are released properly as printed by "Now total ports is 0".

Hi Rahul,

Thanks for detailed explanation.

#6 lore raw
In reply to Rahul Lakkireddy
Quoted 5 lines · Rahul Lakkireddy

Patch 1 fixes queue DMA ring leaks seen during port close.

Patch 2 enables RTE_ETH_DEV_CLOSE_REMOVE and reworks resources release logic for port close.

Quoted 8 lines

Thanks,
Rahul

Rahul Lakkireddy (2):
net/cxgbe: fix queue DMA ring leaks during port close net/cxgbe: release port resources during port close

Series applied to dpdk-next-net/main, thanks.