Skip to contentSkip to conversation

eal: refactor rte_eal_init into sub-functions

PATCH Superseded Rahul Gupta 3 participants 3 messages Jan 19, 2024

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 eal: refactor rte_eal_init into sub-functions

From: Rahul Gupta <[email protected]>

In continuation to the following email, I am sending this patch.
(https://inbox.dpdk.org/dev/[email protected]/)

Initialization requires rte_eal_init + rte_pktmbuf_pool_create which can consume a total time of 500-600 ms:
a) For many devices FLR may take a significant chunk of time (200-250 ms in our use-case), this FLR is triggered during device probe in rte_eal_init().
b) rte_pktmbuf_pool_create() can consume up to 300-350 ms for applications that require huge memory.

This cost is incurred on each restart (which happens in our use-case during binary updates for servicing).
This patch provides an optimization using pthreads that applications can use and which can save 200-230ms.

In this patch, rte_eal_init() is refactored into two parts-
a) 1st part is dependent code ie- it’s a perquisite of the FLR and mempool creation. So this code needs to be executed before any pthreads. Its named as rte_eal_init_setup()
b) 2nd part of code is independent code ie- it can execute in parallel to mempool creation in a pthread. Its named as rte_eal_init_async().

Existing applications requires to just call- rte_eal_init_wait_async_complete() after rte_eal_init() unless they wish to leverage the optimization.

If the application wants to leverage this optimization, then it should create a thread using rte_eal_remote_launch() to schedule a task it would like todo in parallel rte_eal_init_async(), this task can be a mbuf pool creation using- rte_pktmbuf_pool_create()

After both the above tasks, if next operations require completion of above thread a), then user can use rte_eal_init_wait_async_complete(), or if user wants to just check status of that thread, then use- rte_eal_init_async_done()
---
v2: Address Stephen Hemminger's comment
---
v3: address support for single lcore

 app/test-pmd/testpmd.c    | 22 +++++++++++++++++
 lib/eal/include/rte_eal.h | 48 ++++++++++++++++++++++++++++++++++++
 lib/eal/linux/eal.c       | 52 ++++++++++++++++++++++++++++++++++++---
 lib/eal/version.map       |  6 +++++
 4 files changed, 125 insertions(+), 3 deletions(-)
app/test-pmd/testpmd.c +22 -0
index 9e4e99e53b..345f3c1c04 100644
@@ -4531,6 +4531,8 @@ main(int argc, char** argv)
 	portid_t port_id;
 	uint16_t count;
 	int ret;
+	int lcore_id = 0;
+	int main_lcore_id = 0;
 
 #ifdef RTE_EXEC_ENV_WINDOWS
 	signal(SIGINT, signal_handler);
@@ -4555,6 +4557,26 @@ main(int argc, char** argv)
 		rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
 			 rte_strerror(rte_errno));
 
+	main_lcore_id = rte_lcore_id();
+	lcore_id = rte_get_next_lcore(main_lcore_id, 0, 1);
+	/* Gives status of rte_eal_init_async() */
+	if (main_lcore_id != lcore_id)
+		while (rte_eal_init_async_done(lcore_id) == 0)
+			;
+
+	/*
+	 * Use rte_eal_init_wait_async_complete() to get return value of
+	 * rte_eal_init_async().
+	 * Or
+	 * if testpmd application dont want to know progress/status of
+	 * rte_eal_init_async() and just want to wait till it finishes
+	 * then use following function.
+	 */
+	ret = rte_eal_init_wait_async_complete();
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "Cannot init EAL: "
+			 "rte_eal_init_async() failed: %s\n",
+			 strerror(ret));
 	/* allocate port structures, and init them */
 	init_port();
 
lib/eal/include/rte_eal.h +48 -0

54 lines not on this page · raw message

lib/eal/linux/eal.c +49 -3

71 lines not on this page · raw message

lib/eal/version.map +6 -0
index 5e0cd47c82..8cd843109f 100644
@@ -393,6 +393,12 @@ EXPERIMENTAL {
 	# added in 23.07
 	rte_memzone_max_get;
 	rte_memzone_max_set;
+
+	# added in 24.01
+	rte_eal_init_async;
+	rte_eal_init_async_done;
+	rte_eal_init_setup;
+	rte_eal_init_wait_async_complete;
 };
 
 INTERNAL {
Signature · 1 line
2.25.1
#1 lore raw
Quoted 25 lines · Rahul Gupta

From: Rahul Gupta <[email protected]>

In continuation to the following email, I am sending this patch.
(https://inbox.dpdk.org/dev/[email protected]/)

Initialization requires rte_eal_init + rte_pktmbuf_pool_create which can consume a total time of 500-600 ms:
a) For many devices FLR may take a significant chunk of time (200-250 ms in our use-case), this FLR is triggered during device probe in rte_eal_init().
b) rte_pktmbuf_pool_create() can consume up to 300-350 ms for applications that require huge memory.

This cost is incurred on each restart (which happens in our use-case during binary updates for servicing).
This patch provides an optimization using pthreads that applications can use and which can save 200-230ms.

In this patch, rte_eal_init() is refactored into two parts-
a) 1st part is dependent code ie- it’s a perquisite of the FLR and mempool creation. So this code needs to be executed before any pthreads. Its named as rte_eal_init_setup()
b) 2nd part of code is independent code ie- it can execute in parallel to mempool creation in a pthread. Its named as rte_eal_init_async().

Quoted 4 lines

Existing applications requires to just call- rte_eal_init_wait_async_complete() after rte_eal_init() unless they wish to leverage the optimization.

This part here is a bit problematic. For something as ubiquitous as rte_eal_init(), we should not be requiring existing application change. Instead, can you create a new init function to replace rte_eal_init() in the async case - and then eal_init can call that and then the async_complete function, to avoid duplicating code.

The goal should be to have existing apps unaffected, but to allow a path to faster startup for those that need it. In the original email you linked to above, we had such a proposal: rte_eal_init_async() & rte_eal_init_async_done().

Thanks,
/Bruce

#2 lore raw
In reply to Rahul Gupta

app/test-pmd/testpmd.c main(int argc, char** argv) · line 4535

Quoted 10 lines · On Fri, 19 Jan 2024 08:58:17 -0800 Rahul Gupta <[email protected]>…
app/test-pmd/testpmd.c +2 -0
index 9e4e99e53b..345f3c1c04 100644
@@ -4531,6 +4531,8 @@ main(int argc, char** argv)
 	portid_t port_id;
 	uint16_t count;
 	int ret;
+	int lcore_id = 0;
+	int main_lcore_id = 0;

These do not need to be initialized, there is assignment before first usage.

There already exists rte_get_main_lcore() please use that