Skip to contentSkip to conversation

app/testpmd: reduce memory consumption

PATCH Superseded David Marchand 5 participants 20 messages Nov 21, 2019 – Nov 24, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 app/testpmd: reduce memory consumption

Following [1], testpmd memory consumption has skyrocketted. The rte_port structure has gotten quite fat.

struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[65536]; /* 266280 3145728 */
/* --- cacheline 53312 boundary (3411968 bytes) was 40 bytes ago --- */
struct rte_eth_txconf tx_conf[65536]; /* 3412008 3670016 */
/* --- cacheline 110656 boundary (7081984 bytes) was 40 bytes ago --- */
[...]
/* size: 8654936, cachelines: 135234, members: 31 */
[...]

testpmd handles RTE_MAX_ETHPORTS ports (32 by default) which means that it needs ~256MB just for this internal representation.

The reason is that a testpmd rte_port (the name is quite confusing, as it is a local type) maintains configurations for all queues of a port. But where you would expect testpmd to use RTE_MAX_QUEUES_PER_PORT as the maximum queue count, the rte_port uses MAX_QUEUE_ID set to 64k.

Prefer the ethdev maximum value.

After this patch:
struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[1025]; /* 8240 49200 */
/* --- cacheline 897 boundary (57408 bytes) was 32 bytes ago --- */
struct rte_eth_txconf tx_conf[1025]; /* 57440 57400 */
/* --- cacheline 1794 boundary (114816 bytes) was 24 bytes ago --- */
[...]
/* size: 139488, cachelines: 2180, members: 31 */
[...]

[1]: https://git.dpdk.org/dpdk/commit/?id=436b3a6b6e62

---
 app/test-pmd/testpmd.c |  6 +++---
 app/test-pmd/testpmd.h | 16 +++++++---------
 2 files changed, 10 insertions(+), 12 deletions(-)
app/test-pmd/testpmd.c +3 -3
index 73ebf37aae..d28211ba52 100644
@@ -979,7 +979,7 @@ check_socket_id(const unsigned int socket_id)
 queueid_t
 get_allowed_max_nb_rxq(portid_t *pid)
 {
-	queueid_t allowed_max_rxq = MAX_QUEUE_ID;
+	queueid_t allowed_max_rxq = RTE_MAX_QUEUES_PER_PORT;
 	bool max_rxq_valid = false;
 	portid_t pi;
 	struct rte_eth_dev_info dev_info;
@@ -1029,7 +1029,7 @@ check_nb_rxq(queueid_t rxq)
 queueid_t
 get_allowed_max_nb_txq(portid_t *pid)
 {
-	queueid_t allowed_max_txq = MAX_QUEUE_ID;
+	queueid_t allowed_max_txq = RTE_MAX_QUEUES_PER_PORT;
 	bool max_txq_valid = false;
 	portid_t pi;
 	struct rte_eth_dev_info dev_info;
@@ -1079,7 +1079,7 @@ check_nb_txq(queueid_t txq)
 queueid_t
 get_allowed_max_nb_hairpinq(portid_t *pid)
 {
-	queueid_t allowed_max_hairpinq = MAX_QUEUE_ID;
+	queueid_t allowed_max_hairpinq = RTE_MAX_QUEUES_PER_PORT;
 	portid_t pi;
 	struct rte_eth_hairpin_cap cap;
 
app/test-pmd/testpmd.h +7 -9
index 90694a3309..217d577018 100644
@@ -58,8 +58,6 @@ typedef uint16_t portid_t;
 typedef uint16_t queueid_t;
 typedef uint16_t streamid_t;
 
-#define MAX_QUEUE_ID ((1 << (sizeof(queueid_t) * 8)) - 1)
-
 #if defined RTE_LIBRTE_PMD_SOFTNIC
 #define SOFTNIC			1
 #else
@@ -179,22 +177,22 @@ struct rte_port {
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
 	uint8_t                 dcb_flag;   /**< enable dcb */
-	uint16_t                nb_rx_desc[MAX_QUEUE_ID+1]; /**< per queue rx desc number */
-	uint16_t                nb_tx_desc[MAX_QUEUE_ID+1]; /**< per queue tx desc number */
-	struct rte_eth_rxconf   rx_conf[MAX_QUEUE_ID+1]; /**< per queue rx configuration */
-	struct rte_eth_txconf   tx_conf[MAX_QUEUE_ID+1]; /**< per queue tx configuration */
+	uint16_t                nb_rx_desc[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue rx desc number */
+	uint16_t                nb_tx_desc[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue tx desc number */
+	struct rte_eth_rxconf   rx_conf[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue rx configuration */
+	struct rte_eth_txconf   tx_conf[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue tx configuration */
 	struct rte_ether_addr   *mc_addr_pool; /**< pool of multicast addrs */
 	uint32_t                mc_addr_nb; /**< nb. of addr. in mc_addr_pool */
 	uint8_t                 slave_flag; /**< bonding slave port */
 	struct port_flow        *flow_list; /**< Associated flows. */
-	const struct rte_eth_rxtx_callback *rx_dump_cb[MAX_QUEUE_ID+1];
-	const struct rte_eth_rxtx_callback *tx_dump_cb[MAX_QUEUE_ID+1];
+	const struct rte_eth_rxtx_callback *rx_dump_cb[RTE_MAX_QUEUES_PER_PORT+1];
+	const struct rte_eth_rxtx_callback *tx_dump_cb[RTE_MAX_QUEUES_PER_PORT+1];
 #ifdef SOFTNIC
 	struct softnic_port     softport;  /**< softnic params */
 #endif
 	/**< metadata value to insert in Tx packets. */
 	uint32_t		tx_metadata;
-	const struct rte_eth_rxtx_callback *tx_set_md_cb[MAX_QUEUE_ID+1];
+	const struct rte_eth_rxtx_callback *tx_set_md_cb[RTE_MAX_QUEUES_PER_PORT+1];
 };
 
 /**
Signature · 1 line
2.23.0
Acked-by on v1 1/1
#1 lore raw
Quoted 34 lines · David Marchand

Following [1], testpmd memory consumption has skyrocketted. The rte_port structure has gotten quite fat.

struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[65536]; /* 266280 3145728 */
/* --- cacheline 53312 boundary (3411968 bytes) was 40 bytes ago --- */
struct rte_eth_txconf tx_conf[65536]; /* 3412008 3670016 */
/* --- cacheline 110656 boundary (7081984 bytes) was 40 bytes ago --- */
[...]
/* size: 8654936, cachelines: 135234, members: 31 */
[...]

testpmd handles RTE_MAX_ETHPORTS ports (32 by default) which means that it needs ~256MB just for this internal representation.

The reason is that a testpmd rte_port (the name is quite confusing, as it is a local type) maintains configurations for all queues of a port. But where you would expect testpmd to use RTE_MAX_QUEUES_PER_PORT as the maximum queue count, the rte_port uses MAX_QUEUE_ID set to 64k.

Prefer the ethdev maximum value.

After this patch:
struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[1025]; /* 8240 49200 */
/* --- cacheline 897 boundary (57408 bytes) was 32 bytes ago --- */
struct rte_eth_txconf tx_conf[1025]; /* 57440 57400 */
/* --- cacheline 1794 boundary (114816 bytes) was 24 bytes ago --- */
[...]
/* size: 139488, cachelines: 2180, members: 31 */
[...]

Quoted 3 lines

[1]: https://git.dpdk.org/dpdk/commit/?id=436b3a6b6e62

Thanks for figuring this out,

<...>

app/test-pmd/testpmd.h typedef uint16_t portid_t; · line 61

Quoted 9 lines
app/test-pmd/testpmd.h +0 -1
index 90694a3309..217d577018 100644
@@ -58,8 +58,6 @@ typedef uint16_t portid_t;
 typedef uint16_t queueid_t;
 typedef uint16_t streamid_t;
 
-#define MAX_QUEUE_ID ((1 << (sizeof(queueid_t) * 8)) - 1)

No strong opinion, but would it be simpler if assign 'MAX_QUEUE_ID' to 'RTE_MAX_QUEUES_PER_PORT' instead?
#define MAX_QUEUE_ID RTE_MAX_QUEUES_PER_PORT

#2 lore raw
Quoted 8 lines · Ferruh Yigit
Quoted 8 lines
app/test-pmd/testpmd.h +0 -0
index 90694a3309..217d577018 100644
@@ -58,8 +58,6 @@ typedef uint16_t portid_t;
 typedef uint16_t queueid_t;
 typedef uint16_t streamid_t;
 
Quoted 5 lines

app/test-pmd/testpmd.h typedef uint16_t portid_t; · line 61

Quoted 1 line
app/test-pmd/testpmd.h +0 -1
-#define MAX_QUEUE_ID ((1 << (sizeof(queueid_t) * 8)) - 1)

No strong opinion, but would it be simpler if assign 'MAX_QUEUE_ID' to 'RTE_MAX_QUEUES_PER_PORT' instead?
#define MAX_QUEUE_ID RTE_MAX_QUEUES_PER_PORT

This was my first solution once I spotted this.
But I prefer to globally replace: when reading this code, using MAX_QUEUE_ID leaves the impression that testpmd has its own restriction on max queue count.

Signature · 1 line
David Marchand
#3 lore raw
Quoted 16 lines · David Marchand
Quoted 8 lines · Ferruh Yigit
Quoted 8 lines
app/test-pmd/testpmd.h +0 -0
index 90694a3309..217d577018 100644
@@ -58,8 +58,6 @@ typedef uint16_t portid_t;
 typedef uint16_t queueid_t;
 typedef uint16_t streamid_t;
 
Quoted 5 lines

app/test-pmd/testpmd.h typedef uint16_t portid_t; · line 61

Quoted 1 line
app/test-pmd/testpmd.h +0 -1
-#define MAX_QUEUE_ID ((1 << (sizeof(queueid_t) * 8)) - 1)

No strong opinion, but would it be simpler if assign 'MAX_QUEUE_ID' to 'RTE_MAX_QUEUES_PER_PORT' instead?
#define MAX_QUEUE_ID RTE_MAX_QUEUES_PER_PORT

Quoted 4 lines

This was my first solution once I spotted this.
But I prefer to globally replace: when reading this code, using MAX_QUEUE_ID leaves the impression that testpmd has its own restriction on max queue count.

Btw, not sure we want to backport this, or maybe up to branches containing d44f8a485f5d ("app/testpmd: enable per queue configure") Opinions?

Signature · 1 line
David Marchand
#4 lore raw

21/11/2019 17:23, David Marchand:

Quoted 18 lines
Quoted 16 lines · David Marchand
Quoted 13 lines · Ferruh Yigit

app/test-pmd/testpmd.h typedef uint16_t portid_t; · line 61

Quoted 9 lines
app/test-pmd/testpmd.h +0 -1
index 90694a3309..217d577018 100644
@@ -58,8 +58,6 @@ typedef uint16_t portid_t;
 typedef uint16_t queueid_t;
 typedef uint16_t streamid_t;
 
-#define MAX_QUEUE_ID ((1 << (sizeof(queueid_t) * 8)) - 1)

No strong opinion, but would it be simpler if assign 'MAX_QUEUE_ID' to 'RTE_MAX_QUEUES_PER_PORT' instead?
#define MAX_QUEUE_ID RTE_MAX_QUEUES_PER_PORT

Quoted 8 lines
Quoted 4 lines

This was my first solution once I spotted this.
But I prefer to globally replace: when reading this code, using MAX_QUEUE_ID leaves the impression that testpmd has its own restriction on max queue count.

Btw, not sure we want to backport this, or maybe up to branches containing d44f8a485f5d ("app/testpmd: enable per queue configure") Opinions?

I am for not backporting.
It is an optimization (stop wasting some memory).

#5 lore raw
In reply to David Marchand

app/test-pmd/testpmd.h struct rte_port { · line 183

Quoted 8 lines · On Thu, 21 Nov 2019 16:12:55 +0100 David Marchand <[email protected]>…
app/test-pmd/testpmd.h +4 -4
-	uint16_t                nb_rx_desc[MAX_QUEUE_ID+1]; /**< per queue rx desc number */
-	uint16_t                nb_tx_desc[MAX_QUEUE_ID+1]; /**< per queue tx desc number */
-	struct rte_eth_rxconf   rx_conf[MAX_QUEUE_ID+1]; /**< per queue rx configuration */
-	struct rte_eth_txconf   tx_conf[MAX_QUEUE_ID+1]; /**< per queue tx configuration */
+	uint16_t                nb_rx_desc[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue rx desc number */
+	uint16_t                nb_tx_desc[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue tx desc number */
+	struct rte_eth_rxconf   rx_conf[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue rx configuration */
+	struct rte_eth_txconf   tx_conf[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue tx configuration */

Why not put all the per-queue stuff together in one structure and put it at the end. Then dynamically size based on number of queues?

#6 lore raw
Quoted 13 lines · Stephen Hemminger

app/test-pmd/testpmd.h struct rte_port { · line 183

Quoted 8 lines · On Thu, 21 Nov 2019 16:12:55 +0100 David Marchand <[email protected]>…
app/test-pmd/testpmd.h +4 -4
-     uint16_t                nb_rx_desc[MAX_QUEUE_ID+1]; /**< per queue rx desc number */
-     uint16_t                nb_tx_desc[MAX_QUEUE_ID+1]; /**< per queue tx desc number */
-     struct rte_eth_rxconf   rx_conf[MAX_QUEUE_ID+1]; /**< per queue rx configuration */
-     struct rte_eth_txconf   tx_conf[MAX_QUEUE_ID+1]; /**< per queue tx configuration */
+     uint16_t                nb_rx_desc[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue rx desc number */
+     uint16_t                nb_tx_desc[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue tx desc number */
+     struct rte_eth_rxconf   rx_conf[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue rx configuration */
+     struct rte_eth_txconf   tx_conf[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue tx configuration */
Quoted 2 lines

Why not put all the per-queue stuff together in one structure and put it at the end. Then dynamically size based on number of queues?

This is something that could be done.
At first glance, the code is relying on those arrays being contiguous, but it should not be a problem.
The reason for the size '+1' is not obvious to me.
Not saying that would be difficult to investigate and fix/rework all this.

My approach seems the quickest and less risky after rc3. I can look at this post 19.11 (but volunteers are welcome, testpmd needs some love).

Signature · 1 line
David Marchand
Acked-by on v1 1/1
#7 lore raw
In reply to David Marchand

21/11/2019 16:12, David Marchand:

Quoted 34 lines

Following [1], testpmd memory consumption has skyrocketted. The rte_port structure has gotten quite fat.

struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[65536]; /* 266280 3145728 */
/* --- cacheline 53312 boundary (3411968 bytes) was 40 bytes ago --- */
struct rte_eth_txconf tx_conf[65536]; /* 3412008 3670016 */
/* --- cacheline 110656 boundary (7081984 bytes) was 40 bytes ago --- */
[...]
/* size: 8654936, cachelines: 135234, members: 31 */
[...]

testpmd handles RTE_MAX_ETHPORTS ports (32 by default) which means that it needs ~256MB just for this internal representation.

The reason is that a testpmd rte_port (the name is quite confusing, as it is a local type) maintains configurations for all queues of a port. But where you would expect testpmd to use RTE_MAX_QUEUES_PER_PORT as the maximum queue count, the rte_port uses MAX_QUEUE_ID set to 64k.

Prefer the ethdev maximum value.

After this patch:
struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[1025]; /* 8240 49200 */
/* --- cacheline 897 boundary (57408 bytes) was 32 bytes ago --- */
struct rte_eth_txconf tx_conf[1025]; /* 57440 57400 */
/* --- cacheline 1794 boundary (114816 bytes) was 24 bytes ago --- */
[...]
/* size: 139488, cachelines: 2180, members: 31 */
[...]

Quoted 3 lines

[1]: https://git.dpdk.org/dpdk/commit/?id=436b3a6b6e62

I was really concerned by the memory requirement increase due to my patch on ethdev structs.
Thank you for finding these giant arrays.

#8 lore raw

21/11/2019 22:25, Thomas Monjalon:

Quoted 37 lines

21/11/2019 16:12, David Marchand:

Quoted 36 lines

Following [1], testpmd memory consumption has skyrocketted. The rte_port structure has gotten quite fat.

struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[65536]; /* 266280 3145728 */
/* --- cacheline 53312 boundary (3411968 bytes) was 40 bytes ago --- */
struct rte_eth_txconf tx_conf[65536]; /* 3412008 3670016 */
/* --- cacheline 110656 boundary (7081984 bytes) was 40 bytes ago --- */
[...]
/* size: 8654936, cachelines: 135234, members: 31 */
[...]

testpmd handles RTE_MAX_ETHPORTS ports (32 by default) which means that it needs ~256MB just for this internal representation.

The reason is that a testpmd rte_port (the name is quite confusing, as it is a local type) maintains configurations for all queues of a port. But where you would expect testpmd to use RTE_MAX_QUEUES_PER_PORT as the maximum queue count, the rte_port uses MAX_QUEUE_ID set to 64k.

Prefer the ethdev maximum value.

After this patch:
struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[1025]; /* 8240 49200 */
/* --- cacheline 897 boundary (57408 bytes) was 32 bytes ago --- */
struct rte_eth_txconf tx_conf[1025]; /* 57440 57400 */
/* --- cacheline 1794 boundary (114816 bytes) was 24 bytes ago --- */
[...]
/* size: 139488, cachelines: 2180, members: 31 */
[...]

[1]: https://git.dpdk.org/dpdk/commit/?id=436b3a6b6e62

Quoted 7 lines
Quoted 1 line

I was really concerned by the memory requirement increase due to my patch on ethdev structs.
Thank you for finding these giant arrays.

After testing this patch, I realized that you can decrease the memory requirement of test-null.sh from 150 (300 in patch [1]) to only 20 MB.

The following patch [1] was workarounding the big memory requirement by increasing the allocated memory to 300 MB.
[1] https://patches.dpdk.org/patch/63151/

#9 lore raw

PATCH v2 1/1 app/testpmd: reduce memory consumption

In reply to David Marchand

Following [1], testpmd memory consumption has skyrocketted. The rte_port structure has gotten quite fat.

struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[65536]; /* 266280 3145728 */
/* --- cacheline 53312 boundary (3411968 bytes) was 40 bytes ago --- */
struct rte_eth_txconf tx_conf[65536]; /* 3412008 3670016 */
/* --- cacheline 110656 boundary (7081984 bytes) was 40 bytes ago --- */
[...]
/* size: 8654936, cachelines: 135234, members: 31 */
[...]

testpmd handles RTE_MAX_ETHPORTS ports (32 by default) which means that it needs ~256MB just for this internal representation.

The reason is that a testpmd rte_port (the name is quite confusing, as it is a local type) maintains configurations for all queues of a port. But where you would expect testpmd to use RTE_MAX_QUEUES_PER_PORT as the maximum queue count, the rte_port uses MAX_QUEUE_ID set to 64k.

Prefer the ethdev maximum value.

After this patch:
struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[1025]; /* 8240 49200 */
/* --- cacheline 897 boundary (57408 bytes) was 32 bytes ago --- */
struct rte_eth_txconf tx_conf[1025]; /* 57440 57400 */
/* --- cacheline 1794 boundary (114816 bytes) was 24 bytes ago --- */
[...]
/* size: 139488, cachelines: 2180, members: 31 */
[...]

With this, we can ask for less memory in test-null.sh.

[1]: https://git.dpdk.org/dpdk/commit/?id=436b3a6b6e62

---
Changelog since v1:
- updated test-null.sh

---
 app/test-pmd/testpmd.c |  6 +++---
 app/test-pmd/testpmd.h | 16 +++++++---------
 devtools/test-null.sh  |  2 +-
 3 files changed, 11 insertions(+), 13 deletions(-)
app/test-pmd/testpmd.c +3 -3
index 73ebf37aae..d28211ba52 100644
@@ -979,7 +979,7 @@ check_socket_id(const unsigned int socket_id)
 queueid_t
 get_allowed_max_nb_rxq(portid_t *pid)
 {
-	queueid_t allowed_max_rxq = MAX_QUEUE_ID;
+	queueid_t allowed_max_rxq = RTE_MAX_QUEUES_PER_PORT;
 	bool max_rxq_valid = false;
 	portid_t pi;
 	struct rte_eth_dev_info dev_info;
@@ -1029,7 +1029,7 @@ check_nb_rxq(queueid_t rxq)
 queueid_t
 get_allowed_max_nb_txq(portid_t *pid)
 {
-	queueid_t allowed_max_txq = MAX_QUEUE_ID;
+	queueid_t allowed_max_txq = RTE_MAX_QUEUES_PER_PORT;
 	bool max_txq_valid = false;
 	portid_t pi;
 	struct rte_eth_dev_info dev_info;
@@ -1079,7 +1079,7 @@ check_nb_txq(queueid_t txq)
 queueid_t
 get_allowed_max_nb_hairpinq(portid_t *pid)
 {
-	queueid_t allowed_max_hairpinq = MAX_QUEUE_ID;
+	queueid_t allowed_max_hairpinq = RTE_MAX_QUEUES_PER_PORT;
 	portid_t pi;
 	struct rte_eth_hairpin_cap cap;
 
app/test-pmd/testpmd.h +7 -9
index 90694a3309..217d577018 100644
@@ -58,8 +58,6 @@ typedef uint16_t portid_t;
 typedef uint16_t queueid_t;
 typedef uint16_t streamid_t;
 
-#define MAX_QUEUE_ID ((1 << (sizeof(queueid_t) * 8)) - 1)
-
 #if defined RTE_LIBRTE_PMD_SOFTNIC
 #define SOFTNIC			1
 #else
@@ -179,22 +177,22 @@ struct rte_port {
 	uint8_t                 need_reconfig_queues; /**< need reconfiguring queues or not */
 	uint8_t                 rss_flag;   /**< enable rss or not */
 	uint8_t                 dcb_flag;   /**< enable dcb */
-	uint16_t                nb_rx_desc[MAX_QUEUE_ID+1]; /**< per queue rx desc number */
-	uint16_t                nb_tx_desc[MAX_QUEUE_ID+1]; /**< per queue tx desc number */
-	struct rte_eth_rxconf   rx_conf[MAX_QUEUE_ID+1]; /**< per queue rx configuration */
-	struct rte_eth_txconf   tx_conf[MAX_QUEUE_ID+1]; /**< per queue tx configuration */
+	uint16_t                nb_rx_desc[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue rx desc number */
+	uint16_t                nb_tx_desc[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue tx desc number */
+	struct rte_eth_rxconf   rx_conf[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue rx configuration */
+	struct rte_eth_txconf   tx_conf[RTE_MAX_QUEUES_PER_PORT+1]; /**< per queue tx configuration */
 	struct rte_ether_addr   *mc_addr_pool; /**< pool of multicast addrs */
 	uint32_t                mc_addr_nb; /**< nb. of addr. in mc_addr_pool */
 	uint8_t                 slave_flag; /**< bonding slave port */
 	struct port_flow        *flow_list; /**< Associated flows. */
-	const struct rte_eth_rxtx_callback *rx_dump_cb[MAX_QUEUE_ID+1];
-	const struct rte_eth_rxtx_callback *tx_dump_cb[MAX_QUEUE_ID+1];
+	const struct rte_eth_rxtx_callback *rx_dump_cb[RTE_MAX_QUEUES_PER_PORT+1];
+	const struct rte_eth_rxtx_callback *tx_dump_cb[RTE_MAX_QUEUES_PER_PORT+1];
 #ifdef SOFTNIC
 	struct softnic_port     softport;  /**< softnic params */
 #endif
 	/**< metadata value to insert in Tx packets. */
 	uint32_t		tx_metadata;
-	const struct rte_eth_rxtx_callback *tx_set_md_cb[MAX_QUEUE_ID+1];
+	const struct rte_eth_rxtx_callback *tx_set_md_cb[RTE_MAX_QUEUES_PER_PORT+1];
 };
 
 /**
devtools/test-null.sh +1 -1
index 9f9a459f76..6e5b1ad529 100755
@@ -25,6 +25,6 @@ else
 fi
 
 (sleep 1 && echo stop) |
-$testpmd -c $coremask --no-huge -m 150 \
+$testpmd -c $coremask --no-huge -m 20 \
 	$libs --vdev net_null1 --vdev net_null2 $eal_options -- \
 	--no-mlockall --total-num-mbufs=2048 $testpmd_options -ia
Signature · 1 line
2.23.0
#10 lore raw

devtools/test-null.sh else · line 30

Quoted 12 lines · David Marchand
devtools/test-null.sh +1 -1
index 9f9a459f76..6e5b1ad529 100755
@@ -25,6 +25,6 @@ else
 fi
 
 (sleep 1 && echo stop) |
-$testpmd -c $coremask --no-huge -m 150 \
+$testpmd -c $coremask --no-huge -m 20 \
 	$libs --vdev net_null1 --vdev net_null2 $eal_options -- \
 	--no-mlockall --total-num-mbufs=2048 $testpmd_options -ia

What do you think to separate this part, and go with first version of the patchset?

And I am not sure if we should update at all, what is the benefit?

Also script fails after update, because of the additional physical devices and their memory requirement, it is possible to make it run with additional testpmd argument but fails by default.

#11 lore raw

22/11/2019 13:24, Ferruh Yigit:

Quoted 16 lines · David Marchand

devtools/test-null.sh else · line 30

Quoted 12 lines
devtools/test-null.sh +1 -1
index 9f9a459f76..6e5b1ad529 100755
@@ -25,6 +25,6 @@ else
 fi
 
 (sleep 1 && echo stop) |
-$testpmd -c $coremask --no-huge -m 150 \
+$testpmd -c $coremask --no-huge -m 20 \
 	$libs --vdev net_null1 --vdev net_null2 $eal_options -- \
 	--no-mlockall --total-num-mbufs=2048 $testpmd_options -ia

What do you think to separate this part, and go with first version of the patchset?

Quoted 1 line

And I am not sure if we should update at all, what is the benefit?

The benefit it to test that there is no anormal memory needs.

Quoted 3 lines

Also script fails after update, because of the additional physical devices and their memory requirement, it is possible to make it run with additional testpmd argument but fails by default.

This test is supposed to be for null devices only. Why are you trying to add physical devices?
Oh wait, we miss an option -w 0 to be in whitelist mode?

#12 lore raw
Quoted 18 lines · Thomas Monjalon

22/11/2019 13:24, Ferruh Yigit:

Quoted 17 lines · David Marchand

devtools/test-null.sh else · line 30

Quoted 12 lines
devtools/test-null.sh +1 -1
index 9f9a459f76..6e5b1ad529 100755
@@ -25,6 +25,6 @@ else
 fi
 
 (sleep 1 && echo stop) |
-$testpmd -c $coremask --no-huge -m 150 \
+$testpmd -c $coremask --no-huge -m 20 \
 	$libs --vdev net_null1 --vdev net_null2 $eal_options -- \
 	--no-mlockall --total-num-mbufs=2048 $testpmd_options -ia

What do you think to separate this part, and go with first version of the patchset?

And I am not sure if we should update at all, what is the benefit?

Fair enough.

Quoted 11 lines

The benefit it to test that there is no anormal memory needs.

Quoted 3 lines

Also script fails after update, because of the additional physical devices and their memory requirement, it is possible to make it run with additional testpmd argument but fails by default.

This test is supposed to be for null devices only. Why are you trying to add physical devices?
Oh wait, we miss an option -w 0 to be in whitelist mode?

I am not trying to add physical devices, they are already there in my environment, yes adding "-w" can solve it.

#13 lore raw

PATCH 1/1 devtools: disable automatic probing in null testing

The script test-null.sh is supposed to do a quick and simple run of testpmd with null PMD only, for sanity check. As it is not supposed to test probing of any other PMD, physical device probing is switched to whitelist mode by using a fake PCI address (0:0.0).
It will also help to keep memory usage stable across platforms.

---
 devtools/test-null.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
devtools/test-null.sh +1 -1
index 9f9a459f76..d82c6ad193 100755
@@ -26,5 +26,5 @@ fi
 
 (sleep 1 && echo stop) |
 $testpmd -c $coremask --no-huge -m 150 \
-	$libs --vdev net_null1 --vdev net_null2 $eal_options -- \
+	$libs -w 0:0.0 --vdev net_null1 --vdev net_null2 $eal_options -- \
 	--no-mlockall --total-num-mbufs=2048 $testpmd_options -ia
Signature · 1 line
2.23.0
Tested-by on v1 1/1
#14 lore raw

Re: [dpdk-dev] [PATCH] devtools: disable automatic probing in null testing

Quoted 18 lines · Thomas Monjalon

The script test-null.sh is supposed to do a quick and simple run of testpmd with null PMD only, for sanity check. As it is not supposed to test probing of any other PMD, physical device probing is switched to whitelist mode by using a fake PCI address (0:0.0).
It will also help to keep memory usage stable across platforms.

---
 devtools/test-null.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
devtools/test-null.sh +0 -0
index 9f9a459f76..d82c6ad193 100755

devtools/test-null.sh fi · line 30

Quoted 6 lines
devtools/test-null.sh +1 -1
@@ -26,5 +26,5 @@ fi
 
 (sleep 1 && echo stop) |
 $testpmd -c $coremask --no-huge -m 150 \
-	$libs --vdev net_null1 --vdev net_null2 $eal_options -- \
+	$libs -w 0:0.0 --vdev net_null1 --vdev net_null2 $eal_options -- \
 	--no-mlockall --total-num-mbufs=2048 $testpmd_options -ia
author Acked-by
#15 lore raw

Re: [dpdk-dev] [PATCH] devtools: disable automatic probing in null testing

Quoted 20 lines · Ferruh Yigit
Quoted 18 lines · Thomas Monjalon

The script test-null.sh is supposed to do a quick and simple run of testpmd with null PMD only, for sanity check. As it is not supposed to test probing of any other PMD, physical device probing is switched to whitelist mode by using a fake PCI address (0:0.0).
It will also help to keep memory usage stable across platforms.

---
 devtools/test-null.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
devtools/test-null.sh +0 -0
index 9f9a459f76..d82c6ad193 100755
@@ -26,5 +26,5 @@ fi
 
Quoted 8 lines

devtools/test-null.sh fi · line 30

Quoted 6 lines
devtools/test-null.sh +1 -1
 (sleep 1 && echo stop) |
 $testpmd -c $coremask --no-huge -m 150 \
-     $libs --vdev net_null1 --vdev net_null2 $eal_options -- \
+     $libs -w 0:0.0 --vdev net_null1 --vdev net_null2 $eal_options -- \
      --no-mlockall --total-num-mbufs=2048 $testpmd_options -ia

Ugly, but does the job.

Signature · 1 line
David Marchand
#16 lore raw

Re: [dpdk-dev] [PATCH] devtools: disable automatic probing in null testing

22/11/2019 16:56, David Marchand:

Quoted 30 lines · Ferruh Yigit
Quoted 20 lines · Thomas Monjalon
Quoted 18 lines

The script test-null.sh is supposed to do a quick and simple run of testpmd with null PMD only, for sanity check. As it is not supposed to test probing of any other PMD, physical device probing is switched to whitelist mode by using a fake PCI address (0:0.0).
It will also help to keep memory usage stable across platforms.

---
 devtools/test-null.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
devtools/test-null.sh +0 -0
index 9f9a459f76..d82c6ad193 100755
@@ -26,5 +26,5 @@ fi
 
Quoted 8 lines

devtools/test-null.sh fi · line 30

Quoted 6 lines
devtools/test-null.sh +1 -1
 (sleep 1 && echo stop) |
 $testpmd -c $coremask --no-huge -m 150 \
-     $libs --vdev net_null1 --vdev net_null2 $eal_options -- \
+     $libs -w 0:0.0 --vdev net_null1 --vdev net_null2 $eal_options -- \
      --no-mlockall --total-num-mbufs=2048 $testpmd_options -ia
Quoted 2 lines

Ugly, but does the job.

Applied

#17 lore raw

Re: [dpdk-dev] [PATCH] devtools: disable automatic probing in null testing

In reply to Thomas Monjalon

Hi Thomas,

Quoted 7 lines · Thomas Monjalon

The script test-null.sh is supposed to do a quick and simple run of testpmd with null PMD only, for sanity check. As it is not supposed to test probing of any other PMD, physical device probing is switched to whitelist mode by using a fake PCI address (0:0.0).
It will also help to keep memory usage stable across platforms.

With https://patchwork.dpdk.org/patch/62014/, --manual-probe could be used as a more standard way of workarounding the PCI bus.

This is a common issue, we should have cleaner way of addressing it than using hacks relying on the PCI bus not panicking up upon finding an inexistant address. Which is a questionable behavior, users should not be encouraged to rely on it.

Quoted 19 lines
---
 devtools/test-null.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
devtools/test-null.sh +1 -1
index 9f9a459f76..d82c6ad193 100755
@@ -26,5 +26,5 @@ fi
 
 (sleep 1 && echo stop) |
 $testpmd -c $coremask --no-huge -m 150 \
-	$libs --vdev net_null1 --vdev net_null2 $eal_options -- \
+	$libs -w 0:0.0 --vdev net_null1 --vdev net_null2 $eal_options -- \
 	--no-mlockall --total-num-mbufs=2048 $testpmd_options -ia
Signature · 1 line
2.23.0
Signature · 2 lines
Gaëtan Rivet
6WIND
#18 lore raw

Re: [dpdk-dev] [PATCH] devtools: disable automatic probing in null testing

22/11/2019 15:03, Gaëtan Rivet:

Quoted 11 lines

Hi Thomas,

Quoted 7 lines · Thomas Monjalon

The script test-null.sh is supposed to do a quick and simple run of testpmd with null PMD only, for sanity check. As it is not supposed to test probing of any other PMD, physical device probing is switched to whitelist mode by using a fake PCI address (0:0.0).
It will also help to keep memory usage stable across platforms.

Quoted 2 lines

With https://patchwork.dpdk.org/patch/62014/, --manual-probe could be used as a more standard way of workarounding the PCI bus.

I really would like we have a better bus API before implementing some new command line options.
Command line options should be optional and considered only helpers for internal applications.
Anyway it is another discussion, and I hope I will have time and courage to participate in such rework soon.

Quoted 4 lines

This is a common issue, we should have cleaner way of addressing it than using hacks relying on the PCI bus not panicking up upon finding an inexistant address. Which is a questionable behavior, users should not be encouraged to rely on it.

Yes

Applied
#19 lore raw
In reply to David Marchand
Quoted 34 lines · David Marchand

Following [1], testpmd memory consumption has skyrocketted. The rte_port structure has gotten quite fat.

struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[65536]; /* 266280 3145728 */
/* --- cacheline 53312 boundary (3411968 bytes) was 40 bytes ago --- */
struct rte_eth_txconf tx_conf[65536]; /* 3412008 3670016 */
/* --- cacheline 110656 boundary (7081984 bytes) was 40 bytes ago --- */
[...]
/* size: 8654936, cachelines: 135234, members: 31 */
[...]

testpmd handles RTE_MAX_ETHPORTS ports (32 by default) which means that it needs ~256MB just for this internal representation.

The reason is that a testpmd rte_port (the name is quite confusing, as it is a local type) maintains configurations for all queues of a port. But where you would expect testpmd to use RTE_MAX_QUEUES_PER_PORT as the maximum queue count, the rte_port uses MAX_QUEUE_ID set to 64k.

Prefer the ethdev maximum value.

After this patch:
struct rte_port {
[...]
struct rte_eth_rxconf rx_conf[1025]; /* 8240 49200 */
/* --- cacheline 897 boundary (57408 bytes) was 32 bytes ago --- */
struct rte_eth_txconf tx_conf[1025]; /* 57440 57400 */
/* --- cacheline 1794 boundary (114816 bytes) was 24 bytes ago --- */
[...]
/* size: 139488, cachelines: 2180, members: 31 */
[...]

Quoted 7 lines

With this, we can ask for less memory in test-null.sh.

[1]: https://git.dpdk.org/dpdk/commit/?id=436b3a6b6e62

Applied to dpdk-next-net/master, thanks.