service: stop lcore threads before 'finalize'
RFC 1/1 service: stop lcore threads before 'finalize'
I've noticed an occasional segfault from the build system in the service_autotest and after talking with David (CC'd), it seems like it's due to the rte_service_finalize deleting the lcore_states object while active lcores are running.
The below patch is an attempt to solve it by first reassigning all the lcores back to ROLE_RTE before releasing the memory. There is probably a larger question for DPDK proper about actually closing the pending lcore threads, but that's a separate issue. I've been running with the patch for a while, and haven't seen the crash anymore on my system.
Thoughts? Is it acceptable as-is?
---
lib/librte_eal/common/rte_service.c +16 -0
@@ -71,6 +71,8 @@ static struct rte_service_spec_impl *rte_services;static struct core_state *lcore_states;static uint32_t rte_service_library_initialized; +static void service_lcore_uninit (void ); +int32_t rte_service_init (void ) {
@@ -122,6 +124,9 @@ rte_service_finalize(void)if (!rte_service_library_initialized)return ; +/* Ensure that all service threads are returned to the ROLE_RTE +*/ +service_lcore_uninit ();rte_free (rte_services);rte_free (lcore_states);
@@ -897,3 +902,14 @@ rte_service_dump(FILE *f, uint32_t id)return 0 ; } + +static void service_lcore_uninit (void ) +{ +unsigned lcore_id; +RTE_LCORE_FOREACH (lcore_id) { +if (!lcore_states[lcore_id].is_service_core) +continue ; + +while (rte_service_lcore_del (lcore_id) == -EBUSY ); + } +}
---
Quoted 6 lines · Aaron Conole
I've noticed an occasional segfault from the build system in the
service_autotestand after talking with David (CC'd), it seems like it's due to therte_service_finalizedeleting thelcore_statesobject while active lcores are running.
Quoted 7 lines
The below patch is an attempt to solve it by first reassigning all the lcores back to
ROLE_RTEbefore releasing the memory. There is probably a larger question for DPDK proper about actually closing the pending lcore threads, but that's a separate issue. I've been running with the patch for a while, and haven't seen the crash anymore on my system.Thoughts? Is it acceptable as-is?
Added this patch to my env, still reproducing the same issue after ~10-20 tries. I added a breakpoint to service_lcore_uninit that is indeed caught when exiting the test application (just wanted to make sure your change was in my binary).
To reproduce:
I modified app/test/meson.build to have an explicit "-l 0-1" + compiled with your patch.
Then, I started a dummy busyloop "while true; do true; done" in a shell that I had pinned to core 1 (taskset -pc 1 $$). Finally, started another shell (as root), pinned to cores 0-1 on my laptop (taskset -pc 0,1 $$) and ran meson test --gdb --repeat=10000 service_autotest
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff4922700 (LWP 8572)] rte_service_runner_func (arg=<optimized out>) at ../lib/librte_eal/common/rte_service.c:458
458 cs->loops++;
A debugging session is active.
Inferior 1 [process 8566] will be killed.
Quit anyway? (y or n) n
Not confirmed.
Missing separate debuginfos, use: debuginfo-install elfutils-libelf-0.172-2.el7.x86_64 glibc-2.17-260.el7_6.6.x86_64 libgcc-4.8.5-36.el7_6.2.x86_64 libibverbs-17.2-3.el7.x86_64 libnl3-3.2.28-4.el7.x86_64 libpcap-1.5.3-11.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64 openssl-libs-1.0.2k-16.el7_6.1.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) info threads
Id Target Id Frame
* 4 Thread 0x7ffff4922700 (LWP 8572) "lcore-slave-1"rte_service_runner_func (arg=<optimized out>) at ../lib/librte_eal/common/rte_service.c:458
3 Thread 0x7ffff5123700 (LWP 8571) "rte_mp_handle"0x00007ffff63a4b4d in recvmsg () from /lib64/libpthread.so.0
2 Thread 0x7ffff5924700 (LWP 8570) "eal-intr-thread"0x00007ffff60c7603 in epoll_wait () from /lib64/libc.so.6
1 Thread 0x7ffff7fd2c00 (LWP 8566) "dpdk-test" 0x00007ffff7deb96f
in _dl_name_match_p () from /lib64/ld-linux-x86-64.so.2 (gdb) bt
#0 rte_service_runner_func (arg=<optimized out>) at ../lib/librte_eal/common/rte_service.c:458
#1 0x0000000000b2c84f in eal_thread_loop (arg=<optimized out>) at ../lib/librte_eal/linux/eal/eal_thread.c:153
#2 0x00007ffff639ddd5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007ffff60c702d in clone () from /lib64/libc.so.6 (gdb) f 0
#0 rte_service_runner_func (arg=<optimized out>) at ../lib/librte_eal/common/rte_service.c:458
458 cs->loops++;
(gdb) p *cs
$1 = {service_mask = 0, runstate = 0 '\000', is_service_core = 0 '\000', service_active_on_lcore = '\000' <repeats 63 times>, loops = 0, calls_per_service = {0 <repeats 64 times>}}
(gdb) p lcore_config[1]
$2 = {thread_id = 140737296606976, pipe_master2slave = {14, 20}, pipe_slave2master = {21, 22}, f = 0xb26ec0 <rte_service_runner_func>, arg = 0x0, ret = 0, state = RUNNING, socket_id = 0, core_id = 1,
core_index = 1, core_role = 0 '\000', detected = 1 '\001', cpuset =
{__bits = {2, 0 <repeats 15 times>}}}
(gdb) p lcore_config[0]
$3 = {thread_id = 0, pipe_master2slave = {0, 0}, pipe_slave2master = {0, 0}, f = 0x0, arg = 0x0, ret = 0, state = WAIT, socket_id = 0, core_id = 0, core_index = 0, core_role = 0 '\000', detected = 1 '\001',
cpuset = {__bits = {1, 0 <repeats 15 times>}}}
(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff7fd2c00 (LWP 8566))]
#0 0x00007ffff7deb96f in _dl_name_match_p () from /lib64/ld-linux-x86-64.so.2 (gdb) bt
#0 0x00007ffff7deb96f in _dl_name_match_p () from /lib64/ld-linux-x86-64.so.2 #1 0x00007ffff7de4756 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2 #2 0x00007ffff7de4fcf in _dl_lookup_symbol_x () from
/lib64/ld-linux-x86-64.so.2
#3 0x00007ffff7de9d1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2 #4 0x00007ffff7df19da in _dl_runtime_resolve_xsavec () from /lib64/ld-linux-x86-64.so.2
#5 0x00007ffff7deafba in _dl_fini () from /lib64/ld-linux-x86-64.so.2 #6 0x00007ffff6002c29 in __run_exit_handlers () from /lib64/libc.so.6 #7 0x00007ffff6002c77 in exit () from /lib64/libc.so.6 #8 0x00007ffff5feb49c in __libc_start_main () from /lib64/libc.so.6 #9 0x00000000004fa126 in _start ()
--
David Marchand
Quoted 14 lines · David Marchand
Quoted 12 lines · Aaron Conole
I've noticed an occasional segfault from the build system in the
service_autotestand after talking with David (CC'd), it seems like it's due to therte_service_finalizedeleting thelcore_statesobject while active lcores are running.The below patch is an attempt to solve it by first reassigning all the lcores back to
ROLE_RTEbefore releasing the memory. There is probably a larger question for DPDK proper about actually closing the pending lcore threads, but that's a separate issue. I've been running with the patch for a while, and haven't seen the crash anymore on my system.
Quoted 6 lines
Quoted 1 line
Thoughts? Is it acceptable as-is?
Added this patch to my env, still reproducing the same issue after ~10-20 tries. I added a breakpoint to
service_lcore_uninitthat is indeed caught when exiting the test application (just wanted to make sure your change was in my binary).
Harry,
We need a fix for this issue.
Interestingly, Stephen patch that joins all pthreads at rte_eal_cleanup [1] makes this issue disappear.
So my understanding is that we are missing a api (well, I could not find a way) to synchronously stop service lcores.
1: https://patchwork.dpdk.org/patch/64201/
Signature · 1 line
David Marchand
Quoted 18 lines · David Marchand
Quoted 16 lines · David Marchand
Quoted 13 lines · Aaron Conole
I've noticed an occasional segfault from the build system in the
service_autotestand after talking with David (CC'd), it seems like it's due to therte_service_finalizedeleting thelcore_statesobject while active lcores are running.The below patch is an attempt to solve it by first reassigning all the lcores back to
ROLE_RTEbefore releasing the memory. There is probably a larger question for DPDK proper about actually closing the pending lcore threads, but that's a separate issue. I've been running with the patch for a while, and haven't seen the crash anymore on my system.Thoughts? Is it acceptable as-is?
Quoted 8 lines
Quoted 4 lines
Added this patch to my env, still reproducing the same issue after ~10-20 tries. I added a breakpoint to
service_lcore_uninitthat is indeed caught when exiting the test application (just wanted to make sure your change was in my binary).Harry,
We need a fix for this issue.
+1
Quoted 4 lines
Interestingly, Stephen patch that joins all pthreads at
rte_eal_cleanup[1] makes this issue disappear.
So my understanding is that we are missing a api (well, I could not find a way) to synchronously stop service lcores.
Maybe we can take that patch as a fix. I hate to see this segfault in the field. I need to figure out what I missed in my cleanup (probably missed a synchronization point).
Quoted 2 lines
Quoted 34 lines
-----Original Message-----
From: Aaron Conole <[email protected]>
Sent: Tuesday, February 4, 2020 2:51 PM
To: David Marchand <[email protected]>
Cc: Van Haaren, Harry <[email protected]>; dev <[email protected]>
Subject: Re: [RFC] service: stop lcore threads before 'finalize'Quoted 7 lines · David Marchand
Quoted 5 lines · David Marchand
Quoted 3 lines · Aaron Conole
I've noticed an occasional segfault from the build system in the
service_autotestand after talking with David (CC'd), it seems likeit's
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
due to the
rte_service_finalizedeleting thelcore_statesobject while active lcores are running.The below patch is an attempt to solve it by first reassigning all the lcores back to
ROLE_RTEbefore releasing the memory. There is probably a larger question for DPDK proper about actually closing the pending lcore threads, but that's a separate issue. I've been running with the patch for a while, and haven't seen the crash anymore on my system.Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Thoughts? Is it acceptable as-is?
Added this patch to my env, still reproducing the same issue after ~10-20
tries.
Quoted 4 lines
Quoted 3 lines
I added a breakpoint to
service_lcore_uninitthat is indeed caught when exiting the test application (just wanted to make sure your change was in my binary).
Quoted 5 lines
Quoted 3 lines
Harry,
We need a fix for this issue.
+1
Hi All,
Quoted 8 lines
Quoted 4 lines
Interestingly, Stephen patch that joins all pthreads at
rte_eal_cleanup[1] makes this issue disappear.
So my understanding is that we are missing a api (well, I could not find a way) to synchronously stop service lcores.Maybe we can take that patch as a fix. I hate to see this segfault in the field. I need to figure out what I missed in my cleanup (probably missed a synchronization point).
I haven't easily reproduced this yet - so I'll investigate a way to reproduce with close to 100% rate, then we can identify the root cause and actually get a clean fix. If you have pointers to reproduce easily, please let me know.
-H
Quoted 1 line
Quoted 1 line
Quoted 4 lines · Harry
I haven't easily reproduced this yet - so I'll investigate a way to reproduce with close to 100% rate, then we can identify the root cause and actually get a clean fix. If you have pointers to reproduce easily, please let me know.
- In shell #1:
$ git reset --hard v20.02-rc2 HEAD is now at 2636c2a23 version: 20.02-rc2 $ rm -rf build
$ git diff
app/test/meson.build +1 -1
@@ -400,7 +400,7 @@ timeout_seconds = 600 timeout_seconds_fast = 10 get_coremask = find_program('get-coremask.sh') -num_cores_arg = '-l ' + run_command(get_coremask).stdout().strip() +num_cores_arg = '-l 0,1' test_args = [num_cores_arg] foreach arg : fast_test_names
$ meson --werror --buildtype=debugoptimized build The Meson build system Version: 0.47.2 Source dir: /home/dmarchan/dpdk Build dir: /home/dmarchan/dpdk/build Build type: native build Program cat found: YES (/usr/bin/cat) Project name: DPDK Project version: 20.02.0-rc2 ...
$ ninja-build -C build ninja: Entering directory `build' [2081/2081] Linking target app/test/dpdk-test.
$ taskset -pc 1 $$ pid 11143's current affinity list: 0-7 pid 11143's new affinity list: 1
$ while true; do true; done
- Now, in shell #2, as root:
# taskset -pc 0,1 $$ pid 22233's current affinity list: 0-7 pid 22233's new affinity list: 0,1
# meson test --gdb --repeat=10000 service_autotest ...
+ ------------------------------------------------------- +
+ Test Suite Summary
+ Tests Total : 16
+ Tests Skipped : 3
+ Tests Executed : 16
+ Tests Unsupported: 0
+ Tests Passed : 13
+ Tests Failed : 0
+ ------------------------------------------------------- +
Test OK
RTE>>
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff4922700 (LWP 31194)] rte_service_runner_func (arg=<optimized out>) at ../lib/librte_eal/common/rte_service.c:453
453 cs->loops++;
A debugging session is active.
Inferior 1 [process 31187] will be killed.
Quit anyway? (y or n)
I get the crash in like 30s, often less.
In my test right now, I got the crash on the 3rd try.
Signature · 1 line
David Marchand
Quoted 9 lines · Harry
Quoted 9 lines
Quoted 1 line
We need a fix for this issue.
+1
Quoted 4 lines
Interestingly, Stephen patch that joins all pthreads at
rte_eal_cleanup[1] makes this issue disappear.
So my understanding is that we are missing a api (well, I could not find a way) to synchronously stop service lcores.
Quoted 9 lines
Quoted 3 lines
Maybe we can take that patch as a fix. I hate to see this segfault in the field. I need to figure out what I missed in my cleanup (probably missed a synchronization point).
I haven't easily reproduced this yet - so I'll investigate a way to reproduce with close to 100% rate, then we can identify the root cause and actually get a clean fix. If you have pointers to reproduce easily, please let me know.
ping.
I want a fix in 20.05, or I will start considering how to drop this thing.
Signature · 1 line
David Marchand
Quoted 22 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Thursday, February 20, 2020 1:25 PM
To: Van Haaren, Harry <[email protected]>
Cc: Aaron Conole <[email protected]>; dev <[email protected]>
Subject: Re: [RFC] service: stop lcore threads before 'finalize'Quoted 13 lines · Harry
Quoted 12 lines
Quoted 1 line
We need a fix for this issue.
+1
Quoted 4 lines
Interestingly, Stephen patch that joins all pthreads at
rte_eal_cleanup[1] makes this issue disappear.
So my understanding is that we are missing a api (well, I could not find a way) to synchronously stop service lcores.Maybe we can take that patch as a fix. I hate to see this segfault in the field. I need to figure out what I missed in my cleanup (probably missed a synchronization point).
Quoted 8 lines
Quoted 5 lines
I haven't easily reproduced this yet - so I'll investigate a way to reproduce with close to 100% rate, then we can identify the root cause and actually get a clean fix. If you have pointers to reproduce easily, please let me know.
ping.
I want a fix in 20.05, or I will start considering how to drop this thing.
Hi David,
I have been attempting to reproduce, unfortunately without success.
Attempted you suggested meson test approach (thanks for suggesting!), but I haven't had a segfault with that approach (yet, and its done a lot of iterations..)
I've made the service-cores unit tests delay before exit, in an attempt to have them access previously rte_free()-ed memory, no luck to reproduce.
Thinking perhaps we need it on exit, I've also POCed a unit test that leaves service cores active on exit on purpose, to try have them poll after exit, still no luck.
Simplifying the problem, and using hello-world sample app with a rte_eal_cleaup() call at the end also doesn't easily aggravate the problem.
From code inspection, I agree there is an issue. It seems like a call to rte_service_lcore_reset_all() from rte_service_finalize() is enough... But without reproducing it is hard to have good confidence in a fix.
If you have cycles to help, investigating if the above reset_all() call fixes there? Otherwise I'll continue trying to reproduce reliably.
Regards, -HvH
Quoted 32 lines · Harry
Quoted 23 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Thursday, February 20, 2020 1:25 PM
To: Van Haaren, Harry <[email protected]>
Cc: Aaron Conole <[email protected]>; dev <[email protected]>
Subject: Re: [RFC] service: stop lcore threads before 'finalize'Quoted 13 lines · Harry
Quoted 12 lines
Quoted 1 line
We need a fix for this issue.
+1
Quoted 4 lines
Interestingly, Stephen patch that joins all pthreads at
rte_eal_cleanup[1] makes this issue disappear.
So my understanding is that we are missing a api (well, I could not find a way) to synchronously stop service lcores.Maybe we can take that patch as a fix. I hate to see this segfault in the field. I need to figure out what I missed in my cleanup (probably missed a synchronization point).
Quoted 8 lines
Quoted 5 lines
I haven't easily reproduced this yet - so I'll investigate a way to reproduce with close to 100% rate, then we can identify the root cause and actually get a clean fix. If you have pointers to reproduce easily, please let me know.
ping.
I want a fix in 20.05, or I will start considering how to drop this thing.
Quoted 6 lines
Hi David,
I have been attempting to reproduce, unfortunately without success.
Attempted you suggested meson test approach (thanks for suggesting!), but I haven't had a segfault with that approach (yet, and its done a lot of iterations..)
I reproduced it on the first try, just now.
Travis catches it every once in a while (look at the ovsrobot).
For the reproduction, this is on my laptop (core i7-8650U), baremetal, no fancy stuff.
FWIW, the cores are ruled by the "powersave" governor.
I can see the frequency oscillates between 3.5GHz and 3.7Ghz while the max frequency is 4.2GHz.
Travis runs virtual machines with 2 cores, and there must be quite some overprovisioning on those servers.
We can expect some cycles being stolen or at least something happening on the various cores.
Quoted 3 lines
I've made the service-cores unit tests delay before exit, in an attempt to have them access previously rte_free()-ed memory, no luck to reproduce.
Ok, let's forget about the segfault, what do you think of the backtrace I caught?
A service lcore thread is still in the service loop. The master thread of the application is in the libc exiting code.
This is what I get in all crashes.
Quoted 11 lines
Thinking perhaps we need it on exit, I've also POCed a unit test that leaves service cores active on exit on purpose, to try have them poll after exit, still no luck.
Simplifying the problem, and using hello-world sample app with a
rte_eal_cleaup()call at the end also doesn't easily aggravate the problem.From code inspection, I agree there is an issue. It seems like a call to
rte_service_lcore_reset_all()fromrte_service_finalize()is enough... But without reproducing it is hard to have good confidence in a fix.
You promised a doc update on the services API.
Thanks.
--
David Marchand
Quoted 9 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Tuesday, March 10, 2020 1:05 PM
To: Van Haaren, Harry <[email protected]>
Cc: Aaron Conole <[email protected]>; dev <[email protected]>
Subject: Re: [RFC] service: stop lcore threads before 'finalize'On Fri, Feb 21, 2020 at 1:28 PM Van Haaren, Harry
<[email protected]> wrote:
<snip>
Quoted 28 lines
Quoted 7 lines
Hi David,
I have been attempting to reproduce, unfortunately without success.
Attempted you suggested meson test approach (thanks for suggesting!), but I haven't had a segfault with that approach (yet, and its done a lot of
iterations..)
I reproduced it on the first try, just now.
Travis catches it every once in a while (look at the ovsrobot).For the reproduction, this is on my laptop (core i7-8650U), baremetal, no fancy stuff.
FWIW, the cores are ruled by the "powersave" governor.
I can see the frequency oscillates between 3.5GHz and 3.7Ghz while the max frequency is 4.2GHz.Travis runs virtual machines with 2 cores, and there must be quite some overprovisioning on those servers.
We can expect some cycles being stolen or at least something happening on the various cores.Quoted 3 lines
I've made the service-cores unit tests delay before exit, in an attempt to have them access previously rte_free()-ed memory, no luck to reproduce.
Quoted 6 lines
Ok, let's forget about the segfault, what do you think of the backtrace I caught?
A service lcore thread is still in the service loop. The master thread of the application is in the libc exiting code.This is what I get in all crashes.
Hi,
I was actually coding up the above as a patch to send to ML for testing. I've tried to reproduce - it doesn't happen here. I don't like sending patches for fixes that I haven't been able to reliably reproduce and fix locally - but in this case there's I don't see any other option.
I'll post the fix patch to the mailing list ASAP, your and Aaron's help in testing would be greatly appreciated.
Quoted 7 lines
Quoted 1 line
Thinking perhaps we need it on exit, I've also POCed a unit test that
leaves
Quoted 4 lines
service cores active on exit on purpose, to try have them poll after exit, still no luck.
Simplifying the problem, and using hello-world sample app with a
rte_eal_cleaup()
Quoted 8 lines
Quoted 5 lines
call at the end also doesn't easily aggravate the problem.
From code inspection, I agree there is an issue. It seems like a call to
rte_service_lcore_reset_all()fromrte_service_finalize()is enough... But without reproducing it is hard to have good confidence in a fix.You promised a doc update on the services API.
Thanks.
Yes, I heard there are some questions around what service cores is useful for. Having reviewed the programmer guide and doxygen of the API, I'm not sure what needs to change. Do you have specific questions you'd like to see addressed here, or what do you feel needs to change?
https://doc.dpdk.org/guides/prog_guide/service_cores.html
http://doc.dpdk.org/api/rte__service_8h.html
Regards, -Harry