eal: refactor rte_eal_init into sub-functions
PATCH v4 1/1 eal: refactor rte_eal_init into sub-functions
From: Rahul Gupta <[email protected]>
In continuation to the following email, I am sending this patch.
(https://inbox.dpdk.org/dev/[email protected]/)
Initialization requires rte_eal_init + rte_pktmbuf_pool_create which can consume a total time of 500-600 ms:
a) For many devices FLR may take a significant chunk of time (200-250 ms in our use-case), this FLR is triggered during device probe in rte_eal_init().
b) rte_pktmbuf_pool_create() can consume up to 300-350 ms for applications that require huge memory.
This cost is incurred on each restart (which happens in our use-case during binary updates for servicing).
This patch provides an optimization using pthreads that applications can use and which can save 200-230ms.
In this patch, rte_eal_init() is refactored into two parts-
a) 1st part is dependent code ie- it’s a perquisite of the FLR and mempool creation. So this code needs to be executed before any pthreads. Its named as rte_eal_init_setup()
b) 2nd part of code is independent code ie- it can execute in parallel to mempool creation in a pthread. Its named as rte_eal_init_async_setup().
In existing applications no changes are required unless they wish to leverage the optimization.
If the application wants to leverage this optimization, then it needs to call rte_eal_init_async() (instead of call rte_eal_init()), then it can create a thread using rte_eal_remote_launch() to schedule a task it would like todo in parallel rte_eal_init_async_setup(), this task can be a mbuf pool creation using- rte_pktmbuf_pool_create()
After this, if next operations require completion of above task, then user can use rte_eal_init_wait_async_setup_complete(),
or if user wants to just check status of that thread, then use- rte_eal_init_async_setup_done()
---
v2: Address Stephen Hemminger's comment
---
v3: address support for single lcore
---
v4: address Brue Richardson and Stephen Hemminger comment
Existing application need not do any changes if bootup optimization is not needed.
app/test-pmd/testpmd.c | 24 ++++++++- lib/eal/include/rte_eal.h | 107 ++++++++++++++++++++++++++++++++++++++ lib/eal/linux/eal.c | 62 ++++++++++++++++++++-- lib/eal/version.map | 7 +++ 4 files changed, 196 insertions(+), 4 deletions(-)
app/test-pmd/testpmd.c +23 -1
@@ -4531,6 +4531,8 @@ main(int argc, char** argv)portid_t port_id;uint16_t count;int ret; +int lcore_id; +int main_lcore_id;#ifdef RTE_EXEC_ENV_WINDOWS signal (SIGINT , signal_handler);
@@ -4550,11 +4552,31 @@ main(int argc, char** argv)rte_exit (EXIT_FAILURE ,"Cannot register log type" );rte_log_set_level (testpmd_logtype,RTE_LOG_DEBUG ); - diag =rte_eal_init (argc, argv); + diag =rte_eal_init_async (argc, argv);if (diag <0 )rte_exit (EXIT_FAILURE ,"Cannot init EAL: %s\n" ,rte_strerror (rte_errno)); + main_lcore_id =rte_get_main_lcore (); + lcore_id =rte_get_next_lcore (main_lcore_id,0 ,1 ); +/* Gives status of rte_eal_init_async() */ +if (main_lcore_id != lcore_id) +while (rte_eal_init_async_setup_done (lcore_id) ==0 ) + ; + +/* +* Use rte_eal_init_wait_async_setup_complete() to get return value of +* rte_eal_init_async(). +* Or +* if testpmd application don't want to know progress/status of +* rte_eal_init_async() and just want to wait till it finishes +* then use following function. +*/ + ret =rte_eal_init_wait_async_setup_complete (); +if (ret <0 ) +rte_exit (EXIT_FAILURE ,"Cannot init EAL: " +"rte_eal_init_async() failed: %s\n" , +strerror (ret));/* allocate port structures, and init them */ init_port ();
lib/eal/include/rte_eal.h +107 -0
113 lines not on this page · raw message
lib/eal/linux/eal.c +59 -3
81 lines not on this page · raw message
lib/eal/version.map +7 -0
@@ -393,6 +393,13 @@ EXPERIMENTAL { # added in 23.07 rte_memzone_max_get; rte_memzone_max_set; + + # added in 24.01 + rte_eal_init_async; + rte_eal_init_setup; + rte_eal_init_async_setup; + rte_eal_init_async_setup_done; + rte_eal_init_wait_async_setup_complete; }; INTERNAL {
Signature · 1 line
2.25.1
lib/eal/linux/eal.c rte_eal_init(int argc, char **argv) · line 1343
Quoted 10 lines · On Wed, 24 Jan 2024 05:45:11 -0800 Rahul Gupta <[email protected]>…
lib/eal/linux/eal.c +10
-0+ +/* +* waits until function executing on given lcore finishes. +* returns value returned by the function executing on that lcore. +*/ +__rte_experimental int +rte_eal_init_wait_async_setup_complete (void ) +{ +int lcore_id =-1 ; + lcore_id =rte_lcore_id ();
Useless initialization.
lib/eal/linux/eal.c rte_eal_init(int argc, char **argv)
Quoted 6 lines · On Wed, 24 Jan 2024 05:45:11 -0800 Rahul Gupta <[email protected]>…
+ * returns current status of execution on a given lcore
+ */lib/eal/linux/eal.c +4
-0+__rte_experimental int +rte_eal_init_async_setup_done (int lcore_id) +{ +int ret = (lcore_config[lcore_id].state)
parenthesis not needed here
Quoted 35 lines · Rahul Gupta
From: Rahul Gupta <[email protected]>
In continuation to the following email, I am sending this patch.
(https://inbox.dpdk.org/dev/[email protected]/)Initialization requires
rte_eal_init+rte_pktmbuf_pool_createwhich can consume a total time of 500-600 ms:
a) For many devices FLR may take a significant chunk of time (200-250 ms in our use-case), this FLR is triggered during device probe inrte_eal_init().
b)rte_pktmbuf_pool_create()can consume up to 300-350 ms for applications that require huge memory.This cost is incurred on each restart (which happens in our use-case during binary updates for servicing).
This patch provides an optimization using pthreads that applications can use and which can save 200-230ms.In this patch,
rte_eal_init()is refactored into two parts-
a) 1st part is dependent code ie- it’s a perquisite of the FLR and mempool creation. So this code needs to be executed before any pthreads. Its named asrte_eal_init_setup()
b) 2nd part of code is independent code ie- it can execute in parallel to mempool creation in a pthread. Its named asrte_eal_init_async_setup().In existing applications no changes are required unless they wish to leverage the optimization.
If the application wants to leverage this optimization, then it needs to call
rte_eal_init_async()(instead of callrte_eal_init()), then it can create a thread usingrte_eal_remote_launch()to schedule a task it would like todo in parallelrte_eal_init_async_setup(), this task can be a mbuf pool creation using-rte_pktmbuf_pool_create()
Quoted 4 lines
After this, if next operations require completion of above task, then user can use
rte_eal_init_wait_async_setup_complete(),
or if user wants to just check status of that thread, then use-rte_eal_init_async_setup_done()
Looking at what this patch does.. I am under the impression all you really need is rte_eal_init without initial probing.
Such behavior can probably be achieved with a allowlist set to a non existing device (like for example "-a 0000:00:00.0"), then later, use device hotplug.
Some quick note on this patch.
- don't expose symbols externally if they are only for internal use in the same library,
- current version is 24.03, not 24.01 (wrt comment in version.map),
- other OSes are not handled by this patch, please do the work for FreeBSD and Windows,
- as a followup of the previous point, please check if we can share code between OSes and, if so, move the shared code to lib/eal/common,
- did you test this series with multiprocess?
- why should telemetry and other parts of the current rte_eal_init() be left in the second stage of this initialisation pipeline?
Signature · 1 line
David Marchand
Quoted 54 lines · David Marchand
Date: Wed, 24 Jan 2024 16:53:33 +0100
From: David Marchand <[email protected]>
To: Rahul Gupta <[email protected]>
Cc: [email protected], [email protected], [email protected],
[email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], Rahul Gupta <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v4] eal: refactorrte_eal_initinto
sub-functionsQuoted 35 lines · Rahul Gupta
From: Rahul Gupta <[email protected]>
In continuation to the following email, I am sending this patch.
(https://inbox.dpdk.org/dev/[email protected]/)Initialization requires
rte_eal_init+rte_pktmbuf_pool_createwhich can consume a total time of 500-600 ms:
a) For many devices FLR may take a significant chunk of time (200-250 ms in our use-case), this FLR is triggered during device probe inrte_eal_init().
b)rte_pktmbuf_pool_create()can consume up to 300-350 ms for applications that require huge memory.This cost is incurred on each restart (which happens in our use-case during binary updates for servicing).
This patch provides an optimization using pthreads that applications can use and which can save 200-230ms.In this patch,
rte_eal_init()is refactored into two parts-
a) 1st part is dependent code ie- it’s a perquisite of the FLR and mempool creation. So this code needs to be executed before any pthreads. Its named asrte_eal_init_setup()
b) 2nd part of code is independent code ie- it can execute in parallel to mempool creation in a pthread. Its named asrte_eal_init_async_setup().In existing applications no changes are required unless they wish to leverage the optimization.
If the application wants to leverage this optimization, then it needs to call
rte_eal_init_async()(instead of callrte_eal_init()), then it can create a thread usingrte_eal_remote_launch()to schedule a task it would like todo in parallelrte_eal_init_async_setup(), this task can be a mbuf pool creation using-rte_pktmbuf_pool_create()Quoted 4 lines
After this, if next operations require completion of above task, then user can use
rte_eal_init_wait_async_setup_complete(),
or if user wants to just check status of that thread, then use-rte_eal_init_async_setup_done()
Quoted 5 lines
Looking at what this patch does.. I am under the impression all you really need is
rte_eal_initwithout initial probing.
Such behavior can probably be achieved with a allowlist set to a non existing device (like for example "-a 0000:00:00.0"), then later, use device hotplug.
The patch will be useful to all the adapters irrespective of their host plug support.
Quoted 5 lines
Some quick note on this patch.
- don't expose symbols externally if they are only for internal use in the same library,
done in next patch.
Quoted 1 line
- current version is 24.03, not 24.01 (wrt comment in version.map),
done
Quoted 2 lines
- other OSes are not handled by this patch, please do the work for FreeBSD and Windows,
I can send patch to support non-linux OS, but due to lack of setup, I will need help to test same.
Also, I am planning to do the porting at the end (ie after incorporating all review comments, in order to prevent duplication of efforts).
Quoted 2 lines
- as a followup of the previous point, please check if we can share code between OSes and, if so, move the shared code to
lib/eal/common,
The code for rte_eal_init() is different for all three distros, (even if I consider just the 1st part of rte_eal_init() ie rte_eal_init_setup()). So its difficult to move to common dir. We will have todo it separately for all OS.
Quoted 1 line
- did you test this series with multiprocess?
Yes it works fine, I have tested it with simple_mp app.
Quoted 2 lines
- why should telemetry and other parts of the current
rte_eal_init()be left in the second stage of this initialisation pipeline?
Actually motivation of this patch was todo most of the work in parallel ie in second stage, so not only probe/FLR but telemetry and any other work which can be executed in parallel are done here. (pls refer to the link in the commit message for more details)
Quoted 4 lines
Signature · 2 lines
-- David Marchand
Thanks for the reviewing, my comments are inline.
Thanks,
Rahul.
Quoted 7 lines · Rahul Gupta
Quoted 5 lines
Looking at what this patch does.. I am under the impression all you really need is
rte_eal_initwithout initial probing.
Such behavior can probably be achieved with a allowlist set to a non existing device (like for example "-a 0000:00:00.0"), then later, use device hotplug.The patch will be useful to all the adapters irrespective of their host plug support.
I did not say hotplug support is needed.
If what I described already works, this patch adds nothing.
Signature · 1 line
David Marchand
29/01/2024 08:55, David Marchand:
Quoted 7 lines
Quoted 5 lines · Rahul Gupta
Quoted 5 lines
Looking at what this patch does.. I am under the impression all you really need is
rte_eal_initwithout initial probing.
Such behavior can probably be achieved with a allowlist set to a non existing device (like for example "-a 0000:00:00.0"), then later, use device hotplug.
Quoted 5 lines
Quoted 2 lines
The patch will be useful to all the adapters irrespective of their host plug support.
I did not say hotplug support is needed.
If what I described already works, this patch adds nothing.
I agree with David.
Disabling initial probing should provide what you want. Did you test his proposal?
Quoted 23 lines · Thomas Monjalon
Date: Fri, 02 Feb 2024 11:21:59 +0100
From: Thomas Monjalon <[email protected]>
To: Rahul Gupta <[email protected]>
Cc: David Marchand <[email protected]>, [email protected],
[email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: [dpdk-dev] [PATCH v4] eal: refactorrte_eal_initinto
sub-functions29/01/2024 08:55, David Marchand:
Quoted 10 lines
Quoted 7 lines · Rahul Gupta
Quoted 5 lines
Looking at what this patch does.. I am under the impression all you really need is
rte_eal_initwithout initial probing.
Such behavior can probably be achieved with a allowlist set to a non existing device (like for example "-a 0000:00:00.0"), then later, use device hotplug.The patch will be useful to all the adapters irrespective of their host plug support.
Quoted 8 lines
Quoted 2 lines
I did not say hotplug support is needed.
If what I described already works, this patch adds nothing.I agree with David.
Disabling initial probing should provide what you want. Did you test his proposal?
Yes, I was about to reply after testing same, will be done with testing in few days. But I think the bootup time saved by my patch and hot plug patch will be almost same, because apart from FLR (probe()) the extra work done by my patch (i.e. telemetry, rte_service_init() in parallel to mbuf pool creation) are consuming very less bootup time. So in future if more things are added to 2nd part of eal_init (i.e. rte_eal_init_async_setup()), then the bootup time will be less if we use my patch.
I think we can defer this patch till then.
Thanks,
Rahul.
Quoted 53 lines · Rahul Gupta
Date: Sat, 3 Feb 2024 04:57:49 -0800
From: Rahul Gupta <[email protected]>
To: Thomas Monjalon <[email protected]>
Cc: David Marchand <[email protected]>, [email protected],
[email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: [dpdk-dev] [PATCH v4] eal: refactorrte_eal_initinto
sub-functions
User-Agent: Mutt/1.5.21 (2010-09-15)Quoted 23 lines · Thomas Monjalon
Date: Fri, 02 Feb 2024 11:21:59 +0100
From: Thomas Monjalon <[email protected]>
To: Rahul Gupta <[email protected]>
Cc: David Marchand <[email protected]>, [email protected],
[email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: [dpdk-dev] [PATCH v4] eal: refactorrte_eal_initinto
sub-functions29/01/2024 08:55, David Marchand:
Quoted 10 lines
Quoted 7 lines · Rahul Gupta
Quoted 5 lines
Looking at what this patch does.. I am under the impression all you really need is
rte_eal_initwithout initial probing.
Such behavior can probably be achieved with a allowlist set to a non existing device (like for example "-a 0000:00:00.0"), then later, use device hotplug.The patch will be useful to all the adapters irrespective of their host plug support.
Quoted 8 lines
Quoted 2 lines
I did not say hotplug support is needed.
If what I described already works, this patch adds nothing.I agree with David.
Disabling initial probing should provide what you want. Did you test his proposal?Yes, I was about to reply after testing same, will be done with testing in few days. But I think the bootup time saved by my patch and hot plug patch will be almost same, because apart from FLR (
probe()) the extra work done by my patch (i.e. telemetry,rte_service_init()in parallel to mbuf pool creation) are consuming very less bootup time. So in future if more things are added to 2nd part ofeal_init(i.e.rte_eal_init_async_setup()), then the bootup time will be less if we use my patch.
I think we can defer this patch till then.
Quoted 2 lines
Thanks,
Rahul.
Initial tests looks ok wrt application bootup time saving by using the hot plug APIs and so we may not need the patch for rte_eal_init().
Thanks for revewing patch and suggestions.
Thanks,
Rahul.
Tyler Retzlaff
NAKQuoted 47 lines · Rahul Gupta
From: Rahul Gupta <[email protected]>
In continuation to the following email, I am sending this patch.
(https://inbox.dpdk.org/dev/[email protected]/)Initialization requires
rte_eal_init+rte_pktmbuf_pool_createwhich can consume a total time of 500-600 ms:
a) For many devices FLR may take a significant chunk of time (200-250 ms in our use-case), this FLR is triggered during device probe inrte_eal_init().
b)rte_pktmbuf_pool_create()can consume up to 300-350 ms for applications that require huge memory.This cost is incurred on each restart (which happens in our use-case during binary updates for servicing).
This patch provides an optimization using pthreads that applications can use and which can save 200-230ms.In this patch,
rte_eal_init()is refactored into two parts-
a) 1st part is dependent code ie- it’s a perquisite of the FLR and mempool creation. So this code needs to be executed before any pthreads. Its named asrte_eal_init_setup()
b) 2nd part of code is independent code ie- it can execute in parallel to mempool creation in a pthread. Its named asrte_eal_init_async_setup().In existing applications no changes are required unless they wish to leverage the optimization.
If the application wants to leverage this optimization, then it needs to call
rte_eal_init_async()(instead of callrte_eal_init()), then it can create a thread usingrte_eal_remote_launch()to schedule a task it would like todo in parallelrte_eal_init_async_setup(), this task can be a mbuf pool creation using-rte_pktmbuf_pool_create()After this, if next operations require completion of above task, then user can use
rte_eal_init_wait_async_setup_complete(),
or if user wants to just check status of that thread, then use-rte_eal_init_async_setup_done()---
v2: Address Stephen Hemminger's comment
---
v3: address support for single lcore
---
v4: address Brue Richardson and Stephen Hemminger comment
Existing application need not do any changes if bootup optimization is not needed.
Quoted 5 lines
app/test-pmd/testpmd.c | 24 ++++++++- lib/eal/include/rte_eal.h | 107 ++++++++++++++++++++++++++++++++++++++ lib/eal/linux/eal.c | 62 ++++++++++++++++++++-- lib/eal/version.map | 7 +++ 4 files changed, 196 insertions(+), 4 deletions(-)
NAK
changes to EAL need to cover all support OS/platforms, EAL is not Linux-only library.