remove smp barriers in app/test
cover remove smp barriers in app/test
For smp barriers in app/test, remove the unnecessary barriers and fix some bugs.
Feifei Wang (2):
app/test: remove unnecessary barriers for ring stress test app/test: collect perf data after worker threads exit
app/test/test_ring_stress_impl.h | 5 +---- app/test/test_trace_perf.c | 5 ++--- 2 files changed, 3 insertions(+), 7 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/2 app/test: remove unnecessary barriers for ring stress test
The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd == WRK_CMD_RUN", they start running and when worker lcores load "wrk_cmd == WRK_CMD_STOP", they stop.
For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.
For the rmb in test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to wait wrk_cmd to be loaded, then the parameters can be loaded, So the rmb can be removed.
In the meanwhile, fix a typo. The note above storing "stop" into "wrk_cmd" should be "stop test" rather than "start test".
--- app/test/test_ring_stress_impl.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
app/test/test_ring_stress_impl.h +1 -4
@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname, int32_t prcs)fill_ring_elm (&loc_elm, lc);while (wrk_cmd !=WRK_CMD_RUN ) { -rte_smp_rmb ();rte_pause (); }
@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))/* signal worker to start test */ wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();usleep (run_time *US_PER_S ); -/* signal worker to start test */ +/* signal worker to stop test */ wrk_cmd =WRK_CMD_STOP ; -rte_smp_wmb ();/* wait for workers and collect stats. */ mc =rte_lcore_id ();
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
Hi Feifei,
Quoted 8 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==WRK_CMD_RUN", they start running and when worker lcores load "wrk_cmd==WRK_CMD_STOP", they stop.For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.
I think there is a bug in my original code, we should do smp_wmb() *before* setting wrk_cmd, not after:
/* launch on all workers */
RTE_LCORE_FOREACH_WORKER(lc) {
arg[lc].rng = r;
arg[lc].stats = init_stat;
rte_eal_remote_launch(test, &arg[lc], lc);
}
/* signal worker to start test */
+ rte_smp_wmb();
wrk_cmd = WRK_CMD_RUN;- rte_smp_wmb();
usleep(run_time * US_PER_S);
I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd = WRK_CMD_RUN;' But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.
Quoted 3 lines
For the rmb in
test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to waitwrk_cmdto be loaded, then the parameters can be loaded, So the rmb can be removed.
It is not only about parameters loading, it is to prevent worker core to start too early.
As I understand, your goal is to get rid of rte_smp_*() calls. Might be better to replace such places here with _atomic_ semantics. Then, as I can see, we also can get rid of 'volatile' fo wrk_cmd.
Quoted 39 lines
In the meanwhile, fix a typo. The note above storing "stop" into "
wrk_cmd" should be "stop test" rather than "start test".Signed-off-by Feifei Wang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>--- app/test/test_ring_stress_impl.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)app/test/test_ring_stress_impl.h +1
-5@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname, int32_t prcs)fill_ring_elm (&loc_elm, lc);while (wrk_cmd !=WRK_CMD_RUN ) { -rte_smp_rmb ();rte_pause (); }@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))/* signal worker to start test */ wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();usleep (run_time *US_PER_S ); -/* signal worker to start test */ +/* signal worker to stop test */ wrk_cmd =WRK_CMD_STOP ; -rte_smp_wmb ();/* wait for workers and collect stats. */ mc =rte_lcore_id (); --2.17.1
Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
<snip>
Quoted 32 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==WRK_CMD_RUN", theystart
Quoted 1 line
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.I think there is a bug in my original code, we should do
smp_wmb()*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */app/test/test_ring_stress_impl.h +1
-1+rte_smp_wmb (); wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();usleep (run_time *US_PER_S );
Quoted 7 lines
I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=WRK_CMD_RUN;' But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new thread. The rte_smp_rmb() barrier in the thread function is not required as it reads the data that was set before the thread was launched.
I do not know why rte_smp_wmb is required. The update to 'wrk_cmd' is seen by the thread eventually. rte_smp_wmb does not result in update being seen sooner/immediately.
[1] https://www.youtube.com/watch?t=4170&v=KeLBd2EJLOU&feature=youtu.be
Quoted 8 lines
Quoted 4 lines
For the rmb in
test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to waitwrk_cmdto be loaded, then the parameters can be loaded, So the rmb can beremoved.
It is not only about parameters loading, it is to prevent worker core to start too early.
Because 'pthread_launch' provides the 'happens-before' behavior, the worker core will see the updates that happened before the worker was launched.
I suggest changing the commit log to provide the reasoning around pthread_create.
Quoted 46 lines
As I understand, your goal is to get rid of rte_smp_*() calls. Might be better to replace such places here with _atomic_ semantics. Then, as I can see, we also can get rid of 'volatile' fo
wrk_cmd.app/test/test_ring_stress_impl.h test_worker(void *arg, const char *fname, int32_t
Quoted 16 lines
In the meanwhile, fix a typo. The note above storing "stop" into "
wrk_cmd" should be "stop test" rather than "start test".Signed-off-by Feifei Wang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>--- app/test/test_ring_stress_impl.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)app/test/test_ring_stress_impl.h +0
-0@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname, int32_tprcs)
Quoted 24 lines
app/test/test_ring_stress_impl.h +1
-5fill_ring_elm (&loc_elm, lc);while (wrk_cmd !=WRK_CMD_RUN ) { -rte_smp_rmb ();rte_pause (); }@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))/* signal worker to start test */ wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();usleep (run_time *US_PER_S ); -/* signal worker to start test */ +/* signal worker to stop test */ wrk_cmd =WRK_CMD_STOP ; -rte_smp_wmb ();/* wait for workers and collect stats. */ mc =rte_lcore_id (); --2.17.1
Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
Quoted 40 lines
Quoted 33 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==WRK_CMD_RUN", theystart
Quoted 1 line
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.I think there is a bug in my original code, we should do
smp_wmb()*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */app/test/test_ring_stress_impl.h +1
-1+rte_smp_wmb (); wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();usleep (run_time *US_PER_S );Quoted 7 lines
I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=WRK_CMD_RUN;' But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.
Quoted 5 lines
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the new thread. Therte_smp_rmb()barrier in the thread function is not required as it reads the data that was set before the thread was launched.
rte_eal_remote_launch() doesn't call pthread_create().
All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.
Quoted 3 lines
I do not know why
rte_smp_wmbis required. The update to 'wrk_cmd' is seen by the thread eventually.rte_smp_wmbdoes not result in update being seen sooner/immediately.
We don't need it sooner.
We need to make sure it wouldn't be seen by any worker thread before all workers are launched. To make sure all workers start the test at approximately same moment.
That's why I think wmb() should be before 'wrk_cmd = WRK_CMD_RUN;' in my original code.
Quoted 61 lines
[1] https://www.youtube.com/watch?t=4170&v=KeLBd2EJLOU&feature=youtu.be
Quoted 8 lines
Quoted 4 lines
For the rmb in
test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to waitwrk_cmdto be loaded, then the parameters can be loaded, So the rmb can beremoved.
It is not only about parameters loading, it is to prevent worker core to start too early.
Because '
pthread_launch' provides the 'happens-before' behavior, the worker core will see the updates that happened before the worker was launched.I suggest changing the commit log to provide the reasoning around
pthread_create.Quoted 46 lines
As I understand, your goal is to get rid of rte_smp_*() calls. Might be better to replace such places here with _atomic_ semantics. Then, as I can see, we also can get rid of 'volatile' fo
wrk_cmd.app/test/test_ring_stress_impl.h test_worker(void *arg, const char *fname, int32_t
Quoted 16 lines
In the meanwhile, fix a typo. The note above storing "stop" into "
wrk_cmd" should be "stop test" rather than "start test".Signed-off-by Feifei Wang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>--- app/test/test_ring_stress_impl.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)app/test/test_ring_stress_impl.h +0
-0@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname, int32_tprcs)
Quoted 24 lines
app/test/test_ring_stress_impl.h +1
-5fill_ring_elm (&loc_elm, lc);while (wrk_cmd !=WRK_CMD_RUN ) { -rte_smp_rmb ();rte_pause (); }@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))/* signal worker to start test */ wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();usleep (run_time *US_PER_S ); -/* signal worker to start test */ +/* signal worker to stop test */ wrk_cmd =WRK_CMD_STOP ; -rte_smp_wmb ();/* wait for workers and collect stats. */ mc =rte_lcore_id (); --2.17.1
Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
<snip>
Quoted 51 lines
Quoted 7 lines
Quoted 7 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==
WRK_CMD_RUN",Quoted 24 lines
Quoted 24 lines
Quoted 1 line
they
start
Quoted 1 line
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.I think there is a bug in my original code, we should do
smp_wmb()*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */app/test/test_ring_stress_impl.h +1
-1+rte_smp_wmb (); wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();Quoted 8 lines
Quoted 8 lines
usleep(run_time * US_PER_S);I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=
WRK_CMD_RUN;'Quoted 10 lines
Quoted 2 lines
But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the new thread. Therte_smp_rmb()barrier in the thread function is not required as it reads the
Quoted 6 lines
data that was set before the thread was launched.
rte_eal_remote_launch()doesn't callpthread_create().
All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.
Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide behavior similar to pthread_launch and pthread_join respectively.
There is use of rte_smp_*mb in those functions as well. Those need to be fixed first and then look at these.
Quoted 80 lines
Quoted 3 lines
I do not know why
rte_smp_wmbis required. The update to 'wrk_cmd' is seen by the thread eventually.rte_smp_wmbdoes not result in updatebeing seen sooner/immediately.
We don't need it sooner.
We need to make sure it wouldn't be seen by any worker thread before all workers are launched.
To make sure all workers start the test at approximately same moment. That's why I thinkwmb()should be before 'wrk_cmd=WRK_CMD_RUN;' in my original code.Quoted 3 lines
[1]
https://www.youtube.com/watch?t=4170&v=KeLBd2EJLOU&feature=youtu.
beQuoted 10 lines
Quoted 8 lines
Quoted 4 lines
For the rmb in
test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to waitwrk_cmdto be loaded, then the parameters can be loaded, So the rmb can beremoved.
It is not only about parameters loading, it is to prevent worker core to start too early.
Because '
pthread_launch' provides the 'happens-before' behavior, the worker core will see the updates that happened before the worker waslaunched.
Quoted 2 lines
I suggest changing the commit log to provide the reasoning around
pthread_create.Quoted 11 lines
Quoted 11 lines
As I understand, your goal is to get rid of rte_smp_*() calls. Might be better to replace such places here with _atomic_ semantics. Then, as I can see, we also can get rid of 'volatile' fo
wrk_cmd.Quoted 5 lines
In the meanwhile, fix a typo. The note above storing "stop" into "
wrk_cmd" should be "stop test" rather than "start test".Signed-off-by Feifei Wang <[email protected]>Reviewed-by Honnappa NagarahalliQuoted 37 lines
Quoted 37 lines
Quoted 11 lines
Reviewed-by Ruifeng Wang <[email protected]>--- app/test/test_ring_stress_impl.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)app/test/test_ring_stress_impl.h +0
-0@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname,Quoted 1 line
int32_tprcs)
Quoted 24 lines
app/test/test_ring_stress_impl.h +1
-5fill_ring_elm (&loc_elm, lc);while (wrk_cmd !=WRK_CMD_RUN ) { -rte_smp_rmb ();rte_pause (); }@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))/* signal worker to start test */ wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();usleep (run_time *US_PER_S ); -/* signal worker to start test */ +/* signal worker to stop test */ wrk_cmd =WRK_CMD_STOP ; -rte_smp_wmb ();/* wait for workers and collect stats. */ mc =rte_lcore_id (); --2.17.1
Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
Quoted 55 lines · On Fri, 29 Jan 2021 03:17:50 +0000 Honnappa Nagarahalli…
<snip>
Quoted 53 lines
Quoted 7 lines
Quoted 7 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==
WRK_CMD_RUN",Quoted 24 lines
Quoted 24 lines
Quoted 1 line
they
start
Quoted 1 line
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.I think there is a bug in my original code, we should do
smp_wmb()*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */app/test/test_ring_stress_impl.h +1
-1+rte_smp_wmb (); wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();Quoted 8 lines
Quoted 8 lines
usleep(run_time * US_PER_S);I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=
WRK_CMD_RUN;'Quoted 10 lines
Quoted 2 lines
But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the new thread. Therte_smp_rmb()barrier in the thread function is not required as it reads thedata that was set before the thread was launched.
Quoted 8 lines
Quoted 4 lines
rte_eal_remote_launch()doesn't callpthread_create().
All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.Thanks for the reminder ☹
I thinkrte_eal_remote_launchandrte_eal_wait_lcoreneed to provide behavior similar topthread_launchandpthread_joinrespectively.There is use of rte_smp_*mb in those functions as well. Those need to be fixed first and then look at these.
Looks like you want __atomic_thread_fence() here.
Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
<snip>
Quoted 63 lines
Quoted 43 lines
Quoted 43 lines
Quoted 8 lines
Quoted 8 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==
WRK_CMD_RUN",Quoted 34 lines
Quoted 34 lines
Quoted 1 line
they
start
Quoted 1 line
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.I think there is a bug in my original code, we should do
smp_wmb()
*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */app/test/test_ring_stress_impl.h +1
-1+rte_smp_wmb (); wrk_cmd =WRK_CMD_RUN ; -rte_smp_wmb ();usleep (run_time *US_PER_S );I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=Quoted 3 lines
Quoted 3 lines
WRK_CMD_RUN;'Quoted 2 lines
Quoted 2 lines
But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical
pass.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the newthread.
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
The
rte_smp_rmb()barrier in the thread function is not required as it reads thedata that was set before the thread was launched.
rte_eal_remote_launch()doesn't callpthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.
Quoted 9 lines
Quoted 2 lines
Thanks for the reminder ☹
I thinkrte_eal_remote_launchandrte_eal_wait_lcoreneed to providebehavior similar to
pthread_launchandpthread_joinrespectively.Quoted 2 lines
There is use of rte_smp_*mb in those functions as well. Those need to be fixed
first and then look at these.
Looks like you want
__atomic_thread_fence()here.
In the rte_eal_remote_launch case, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In the eal_thread_loop function we could do load-acquire on the function pointer.
I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).
The rte_eal_wait_lcore API needs to ensure that the memory operations in the worker are visible to the main. rte_eal_wait_lcore and eal_thread_loop are synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' in eal_thread_loop and a load-acquire in rte_eal_wait_lcore.
So, we do not have to use the __atomic_thread_fence.
[dpdk-dev] 回复: [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
Hi, Honnappa, Konstantin and Stephen
Thanks very much for your attention of this patch. Based on your opinion, Ruifeng and I discuss about this and make a summary:
____________________________________________________________________________________________________________________
main thread worker threadrte_eal_remote_launch: [ Honnappa focus ]
To ensure f can load correct arg,
arg store should before f
lcore_config[worker_id].f = f;
lcore_config[worker_id].arg = arg;
wmb()? or store-relase on f?
eal_thread_loop:
pipeline_communication ----------------------> pipeline_communication
if (lcore_config[lcore_id].f == NULL)
rte_panic("NULL function pointer\n");
fct_arg = lcore_config[lcore_id].arg;
ret = lcore_config[lcore_id].f(fct_arg);___________________________________________________________________________________________________________________
test_ring_stress: wmb()? [ Konstantin focus ]
test_worker: Main thread can use wrk_cmd to Wrk_cmd =WRK_CMD_RUN; ----------------------> Wrk_cmd == WRK_CMD_RUN; control multiple threads to start running wmb()? at the same time as much as possible ring_dequeue; ring_enqueue; Wrk_cmd =WRK_CMD_STOP; ----------------------> Wrk_cmd == WRK_CMD_STOP;
____________________________________________________________________________________________________________________
rte_eal_wait_lcore: wmb() [ Honnappa focus ]
lcore_config[lcore_id].state == FINISHED <--------------------- lcore_config[lcore_id].state = FINISHED Load-acquire and store-release are used on the variable “state” rmb();
____________________________________________________________________________________________________________________
From the picture above,
1.First, for the underlying function rte_eal_remote_launch, Honnappa focuses on that, pipeline_communication cannot ensure ‘arg’ parameters is loaded correctly by the worker thread.
This is because in weak memory order framework, maybe the main thread and worker thread firstly finish pipeline communication, and then the worker thread receive signal and execute the function ‘ f ’. However, it maybe load a wrong value of ‘arg’ due to that the main thread stores ‘arg’ after pipeline communication. So wmb or store_release is necessary for ‘arg’.
2.Second, for the upper-layer test_ring_stress, Konstantin foucese on that, Whether the main thread can use ‘wrk_cmd’ to control multiple threads to run at the same time as much as possible.
Because rte_eal_remote_launch only can communicates with one worker thread
at the same time. This means some worker thread maybe start working very early but other worker threads maybe need to wait a long time to start working if ‘wrk_cmd' is stored 'RUN' flag before rte_remote_launch.
At last, for unit test, this may cause that the test results are not stable.
3.Third, for rte_eal_wait_lcore, Honnappa focuses on that the ‘state’ as a synchronous bariable,
we should add load-acquire and store-release on it. However, there have been rmb and wmb after and before ‘state’, So I’m not sure whether we should replace them.
In summary, I think Honnappa and Konstantin have different concerns.
For Honnappa, we can add wmb or store-release to ensure the ‘arg’ can be loaded correctly in rte_eal_remote_launch.
For Konstantin, we can add wmb and rmb to ensure the main thread can control the worker Threads to run at the same time, and then make the test results more accurate in the ring_stress_test.
Best Regards
Feifei
Quoted 109 lines
-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test<snip>
Quoted 68 lines
Quoted 44 lines
Quoted 44 lines
Quoted 8 lines
Quoted 8 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==
WRK_CMD_RUN",Quoted 35 lines
Quoted 35 lines
Quoted 1 line
they
start
Quoted 2 lines
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.I think there is a bug in my original code, we should do
smp_wmb()
*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */(quoted diff) +1
-1+ rte_smp_wmb(); wrk_cmd = WRK_CMD_RUN; - rte_smp_wmb(); usleep(run_time * US_PER_S);I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=Quoted 4 lines
Quoted 4 lines
WRK_CMD_RUN;'Quoted 3 lines
Quoted 3 lines
But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical
pass.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the newthread.
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
The
rte_smp_rmb()barrier in the thread function is not required as it reads thedata that was set before the thread was launched.
rte_eal_remote_launch()doesn't callpthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.Thanks for the reminder ☹
I thinkrte_eal_remote_launchandrte_eal_wait_lcoreneed to provideQuoted 8 lines
behavior similar to
pthread_launchandpthread_joinrespectively.Quoted 3 lines
There is use of rte_smp_*mb in those functions as well. Those need to be fixed
first and then look at these.
Looks like you want
__atomic_thread_fence()here.In the
rte_eal_remote_launchcase, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In theeal_thread_loopfunction we could do load-acquire on the function pointer.I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).
The
rte_eal_wait_lcoreAPI needs to ensure that the memory operations in the worker are visible to the main.rte_eal_wait_lcoreandeal_thread_loopare synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' ineal_thread_loopand a load-acquire inrte_eal_wait_lcore.So, we do not have to use the
__atomic_thread_fence.
Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
Hi Feifei,
Quoted 79 lines
Hi, Honnappa, Konstantin and Stephen
Thanks very much for your attention of this patch. Based on your opinion, Ruifeng and I discuss about this and make a summary:
_________________________________________________________________________________________________________________
___
main thread worker threadrte_eal_remote_launch:[ Honnappa focus ] To ensure f can load correct arg, arg store should before f lcore_config[worker_id].f = f; lcore_config[worker_id].arg = arg; wmb()? or store-relase on f? eal_thread_loop: pipeline_communication ----------------------> pipeline_communication if (lcore_config[lcore_id].f ==NULL)
rte_panic("NULL function
pointer\n");
fct_arg=
lcore_config[lcore_id].arg;
ret =
lcore_config[lcore_id].f(fct_arg);
_________________________________________________________________________________________________________________
__
test_ring_stress:wmb()?[ Konstantin focus ] test_worker: Main thread can use wrk_cmd to Wrk_cmd =WRK_CMD_RUN; ----------------------> Wrk_cmd == WRK_CMD_RUN; control multiple threads to start running wmb()? at the same time as much as possible ring_dequeue; ring_enqueue; Wrk_cmd =WRK_CMD_STOP; ----------------------> Wrk_cmd == WRK_CMD_STOP;_________________________________________________________________________________________________________________
___
rte_eal_wait_lcore:wmb()[ Honnappa focus ] lcore_config[lcore_id].state == FINISHED <--------------------- lcore_config[lcore_id].state =FINISHED Load-acquire and store-release
are used on the variable “state” rmb();_________________________________________________________________________________________________________________
___From the picture above,
1.First, for the underlying function
rte_eal_remote_launch, Honnappa focuses on that,pipeline_communicationcannot ensure ‘arg’ parameters is loaded correctly by the worker thread.
This is because in weak memory order framework, maybe the main thread and worker thread firstly finish pipeline communication, and then the worker thread receive signal and execute the function ‘ f ’. However, it maybe load a wrong value of ‘arg’ due to that the main thread stores ‘arg’ after pipeline communication. So wmb orstore_releaseis necessary for ‘arg’.2.Second, for the upper-layer
test_ring_stress, Konstantin foucese on that, Whether the main thread can use ‘wrk_cmd’ to control multiple threads to run at the same time as much as possible.
Becauserte_eal_remote_launchonly can communicates with one worker thread
at the same time. This means some worker thread maybe start working very early but other worker threads maybe need to wait a long time to start working if ‘wrk_cmd' is stored 'RUN' flag beforerte_remote_launch.
At last, for unit test, this may cause that the test results are not stable.3.Third, for
rte_eal_wait_lcore, Honnappa focuses on that the ‘state’ as a synchronous bariable,
we should add load-acquire and store-release on it. However, there have been rmb and wmb after and before ‘state’, So I’m not sure whether we should replace them.
Quoted 6 lines
In summary, I think Honnappa and Konstantin have different concerns.
For Honnappa, we can add wmb or store-release to ensure the ‘arg’ can be loaded correctly inrte_eal_remote_launch.
For Konstantin, we can add wmb and rmb to ensure the main thread can control the worker Threads to run at the same time, and then make the test results more accurate in thering_stress_test.
Agree with both.
Quoted 114 lines
Best Regards
FeifeiQuoted 109 lines
-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test<snip>
Quoted 68 lines
Quoted 44 lines
Quoted 44 lines
Quoted 8 lines
Quoted 8 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==
WRK_CMD_RUN",Quoted 35 lines
Quoted 35 lines
Quoted 1 line
they
start
Quoted 2 lines
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.I think there is a bug in my original code, we should do
smp_wmb()
*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */(quoted diff) +1
-1+ rte_smp_wmb(); wrk_cmd = WRK_CMD_RUN; - rte_smp_wmb(); usleep(run_time * US_PER_S);I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=Quoted 4 lines
Quoted 4 lines
WRK_CMD_RUN;'Quoted 3 lines
Quoted 3 lines
But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical
pass.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the newthread.
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
The
rte_smp_rmb()barrier in the thread function is not required as it reads thedata that was set before the thread was launched.
rte_eal_remote_launch()doesn't callpthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.Thanks for the reminder ☹
I thinkrte_eal_remote_launchandrte_eal_wait_lcoreneed to provideQuoted 8 lines
behavior similar to
pthread_launchandpthread_joinrespectively.Quoted 3 lines
There is use of rte_smp_*mb in those functions as well. Those need to be fixed
first and then look at these.
Looks like you want
__atomic_thread_fence()here.In the
rte_eal_remote_launchcase, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In theeal_thread_loopfunction we could do load-acquire on the function pointer.I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).
The
rte_eal_wait_lcoreAPI needs to ensure that the memory operations in the worker are visible to the main.rte_eal_wait_lcoreandeal_thread_loopare synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' ineal_thread_loopand a load-acquire inrte_eal_wait_lcore.So, we do not have to use the
__atomic_thread_fence.
Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
<snip>
Quoted 115 lines
Hi Feifei,
Quoted 6 lines
Hi, Honnappa, Konstantin and Stephen
Thanks very much for your attention of this patch. Based on your opinion, Ruifeng and I discuss about this and make a summary:
__________________________________________________________
____________Quoted 3 lines
___________________________________________
___
main threadworker thread
Quoted 8 lines
rte_eal_remote_launch:
[ Honnappa focus ]To ensure f can load correct arg,
arg store should before f
lcore_config[worker_id].f = f; lcore_config[worker_id].arg =arg;
Quoted 3 lines
wmb()? or store-relase on f?
eal_thread_loop:Quoted 1 line
pipeline_communication----------------------> pipeline_communicationQuoted 1 line
if (lcore_config[lcore_id].f ==Quoted 2 lines
NULL)
rte_panic("NULL function
Quoted 3 lines
pointer\n");
fct_arg=Quoted 2 lines
lcore_config[lcore_id].arg;
ret =
Quoted 2 lines
lcore_config[lcore_id].f(fct_arg);
__________________________________________________________
____________Quoted 6 lines
___________________________________________
__
test_ring_stress:wmb()?
[ Konstantin focus ]
test_worker:Quoted 2 lines
Main thread can use wrk_cmd to Wrk_cmd =WRK_CMD_RUN;----------------------> Wrk_cmd == WRK_CMD_RUN;Quoted 2 lines
control multiple threads to start running
wmb()?Quoted 2 lines
at the same time as much as possible
ring_dequeue;Quoted 1 line
ring_enqueue;Quoted 1 line
Wrk_cmd =WRK_CMD_STOP;----------------------> Wrk_cmd == WRK_CMD_STOP;Quoted 1 line
__________________________________________________________
____________Quoted 4 lines
___________________________________________
___
rte_eal_wait_lcore:wmb()Quoted 2 lines
[ Honnappa focus ] lcore_config[lcore_id].state == FINISHED<--------------------- lcore_config[lcore_id].state =Quoted 5 lines
FINISHED Load-acquire and store-release
are used on the variable “state” rmb();__________________________________________________________
____________Quoted 5 lines
___________________________________________
___From the picture above,
Quoted 8 lines
1.First, for the underlying function
rte_eal_remote_launch, Honnappa focuses on that,pipeline_communicationcannot ensure ‘arg’ parameters is loaded correctly by the worker thread.
This is because in weak memory order framework, maybe the main thread and worker thread firstly finish pipeline communication, and then the worker thread receive signal and execute the function ‘ f ’. However, it maybe load a wrong value of ‘arg’ due to that the main thread stores ‘arg’ after pipeline communication. So wmb orstore_releaseisnecessary for ‘arg’.
Quoted 8 lines
2.Second, for the upper-layer
test_ring_stress, Konstantin foucese on that, Whether the main thread can use ‘wrk_cmd’ to control multiple threads to run at the same time as much as possible. Becauserte_eal_remote_launchonly can communicates with one worker thread at the same time. This means some worker thread maybe start working very early but other worker threads maybe need to wait a long time to start working if ‘wrk_cmd' is stored 'RUN' flag before
rte_remote_launch.Quoted 3 lines
At last, for unit test, this may cause that the test results are not stable.
3.Third, for
rte_eal_wait_lcore, Honnappa focuses on that the ‘state’ as asynchronous bariable,
Quoted 2 lines
we should add load-acquire and store-release on it. However, there have been rmb and wmb after and before ‘state’, So I’m not sure whether
we should replace them.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
In summary, I think Honnappa and Konstantin have different concerns. For Honnappa, we can add wmb or store-release to ensure the ‘arg’ can be loaded correctly in
rte_eal_remote_launch.
For Konstantin, we can add wmb and rmb to ensure the main thread can control the worker Threads to run at the same time, and then make the test results more accurate in thering_stress_test.Agree with both.
Thanks Feifei, understood. I am just trying to take a step back and see what kind of ordering guarantees rte_eal_remote_launch should provide so that we do not have to deal with adding additional barriers in the applications. For ex: if we can avoid the barriers around 'wrk_cmd' (kind of use cases) it will benefit all the applications.
Quoted 121 lines
Quoted 11 lines
Best Regards
FeifeiQuoted 5 lines
-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]> 抄送: Ananyev, Konstantin <[email protected]>; FeifeiWang
Quoted 1 line
Quoted 1 line
<[email protected]>; [email protected]; nd <[email protected]>; Ruifeng
Wang
Quoted 18 lines
Quoted 18 lines
<[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test<snip>
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
Hi Feifei,
Quoted 4 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==
WRK_CMD_RUN",Quoted 1 line
Quoted 1 line
Quoted 1 line
they
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
start
Quoted 2 lines
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb isunnecessary.
Quoted 71 lines
Quoted 71 lines
Quoted 51 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
I think there is a bug in my original code, we should do
smp_wmb()
*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */(quoted diff) +1
-1+ rte_smp_wmb(); wrk_cmd = WRK_CMD_RUN; - rte_smp_wmb(); usleep(run_time * US_PER_S);I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=Quoted 4 lines
Quoted 4 lines
WRK_CMD_RUN;'Quoted 3 lines
Quoted 3 lines
But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical
pass.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the newthread.
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
The
rte_smp_rmb()barrier in the thread function is not required as it reads thedata that was set before the thread was launched.
Quoted 7 lines
Quoted 4 lines
rte_eal_remote_launch()doesn't callpthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.Thanks for the reminder ☹
I thinkrte_eal_remote_launchandrte_eal_wait_lcoreneed to provideQuoted 8 lines
behavior similar to
pthread_launchandpthread_joinrespectively.Quoted 3 lines
There is use of rte_smp_*mb in those functions as well. Those need to be fixed
first and then look at these.
Looks like you want
__atomic_thread_fence()here.In the
rte_eal_remote_launchcase, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In theeal_thread_loopfunction we could do load-acquire on the function pointer.I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).
Quoted 5 lines
Quoted 5 lines
The
rte_eal_wait_lcoreAPI needs to ensure that the memory operations in the worker are visible to the main.rte_eal_wait_lcoreandeal_thread_loopare synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on'state' in
eal_thread_loopand a load-acquire inrte_eal_wait_lcore.Quoted 3 lines
Quoted 3 lines
So, we do not have to use the
__atomic_thread_fence.
[dpdk-dev] 回复: [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
In reply to Honnappa NagarahalliSorry, a mistake happens in the picture, after Wrk_cmd == WRK_CMD_RUN, it should be a rmb rather than wmb.
Quoted 109 lines
-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test<snip>
Quoted 68 lines
Quoted 44 lines
Quoted 44 lines
Quoted 8 lines
Quoted 8 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==
WRK_CMD_RUN",Quoted 35 lines
Quoted 35 lines
Quoted 1 line
they
start
Quoted 2 lines
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.I think there is a bug in my original code, we should do
smp_wmb()
*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */(quoted diff) +1
-1+ rte_smp_wmb(); wrk_cmd = WRK_CMD_RUN; - rte_smp_wmb(); usleep(run_time * US_PER_S);I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=Quoted 4 lines
Quoted 4 lines
WRK_CMD_RUN;'Quoted 3 lines
Quoted 3 lines
But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical
pass.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the newthread.
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
The
rte_smp_rmb()barrier in the thread function is not required as it reads thedata that was set before the thread was launched.
rte_eal_remote_launch()doesn't callpthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.Thanks for the reminder ☹
I thinkrte_eal_remote_launchandrte_eal_wait_lcoreneed to provideQuoted 8 lines
behavior similar to
pthread_launchandpthread_joinrespectively.Quoted 3 lines
There is use of rte_smp_*mb in those functions as well. Those need to be fixed
first and then look at these.
Looks like you want
__atomic_thread_fence()here.In the
rte_eal_remote_launchcase, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In theeal_thread_loopfunction we could do load-acquire on the function pointer.I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).
The
rte_eal_wait_lcoreAPI needs to ensure that the memory operations in the worker are visible to the main.rte_eal_wait_lcoreandeal_thread_loopare synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' ineal_thread_loopand a load-acquire inrte_eal_wait_lcore.So, we do not have to use the
__atomic_thread_fence.
[dpdk-dev] 回复: [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test
Hi, everyone
Sorry for that there may be a problem in the e-mail format. Please see the picture according to the following link:
https://ibb.co/SQ7yGfW
Best Regards
Feifei
Quoted 127 lines
-----邮件原件-----
发件人: Feifei Wang
发送时间: 2021年2月1日 16:49
收件人: Honnappa Nagarahalli <[email protected]>; Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
主题: 回复: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress testSorry, a mistake happens in the picture, after
Wrk_cmd==WRK_CMD_RUN, it should be a rmb rather than wmb.Quoted 6 lines
-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; [email protected]; nd <[email protected]>; RuifengWang
Quoted 5 lines
<[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress testQuoted 20 lines
<snip>
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
Hi Feifei,
Quoted 3 lines
The variable "
wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd==
WRK_CMD_RUN",Quoted 9 lines
Quoted 9 lines
Quoted 1 line
they
start
Quoted 2 lines
running and when worker lcores load "
wrk_cmd==WRK_CMD_STOP",they
Quoted 4 lines
stop.
For the wmb in
test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb isunnecessary.
Quoted 71 lines
Quoted 51 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
I think there is a bug in my original code, we should do
smp_wmb()
*before* settingwrk_cmd, not after:/* launch on all workers */ RTE_LCORE_FOREACH_WORKER(lc) { arg[lc].rng = r; arg[lc].stats = init_stat; rte_eal_remote_launch(test, &arg[lc], lc); } /* signal worker to start test */(quoted diff) +1
-1+ rte_smp_wmb(); wrk_cmd = WRK_CMD_RUN; - rte_smp_wmb(); usleep(run_time * US_PER_S);I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update
wrk_cmdout of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd=Quoted 4 lines
Quoted 4 lines
WRK_CMD_RUN;'Quoted 3 lines
Quoted 3 lines
But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical
pass.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Agree that this is not performance critical.
This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the
pthread_createhas 'happens-before' behavior. i.e. all the memory operations before thepthread_createare visible to the newthread.
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
The
rte_smp_rmb()barrier in the thread function is not required as it reads thedata that was set before the thread was launched.
Quoted 7 lines
Quoted 4 lines
rte_eal_remote_launch()doesn't callpthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.Thanks for the reminder ☹
I thinkrte_eal_remote_launchandrte_eal_wait_lcoreneed to provideQuoted 8 lines
behavior similar to
pthread_launchandpthread_joinrespectively.Quoted 3 lines
There is use of rte_smp_*mb in those functions as well. Those need to be fixed
first and then look at these.
Looks like you want
__atomic_thread_fence()here.In the
rte_eal_remote_launchcase, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In theeal_thread_loopfunction we could do load-acquire on the function pointer.I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).
Quoted 5 lines
The
rte_eal_wait_lcoreAPI needs to ensure that the memory operations in the worker are visible to the main.rte_eal_wait_lcoreandeal_thread_loopare synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' ineal_thread_loopand a load-acquire in
rte_eal_wait_lcore.Quoted 3 lines
So, we do not have to use the
__atomic_thread_fence.
PATCH 2/2 app/test: collect perf data after worker threads exit
In reply to Feifei WangThe measure_perf function should be excuted after worker threads exit to collect correct perf data. Otherwise, while workers are running, the main thread may get incomplete data from workers.
In the meanwhile, remove unnecessary barrier in the test. For signal variables "ldata.done" and "ldata.start", no operations should keep the order that being executed after them. So the wmb after them can be moved.
--- app/test/test_trace_perf.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
app/test/test_trace_perf.c +2 -3
@@ -79,7 +79,6 @@ signal_workers_to_finish(struct test_data *data)for (workers =0 ; workers < data->nb_workers; workers++) { data->ldata[workers].done =1 ; -rte_smp_wmb (); } }
@@ -102,7 +101,6 @@ worker_fn_##func(void *arg) \ { \struct lcore_data *ldata = arg; \ ldata->started =1 ; \ -rte_smp_wmb (); \ __worker_##func(ldata); \ return 0; \ }
@@ -137,11 +135,12 @@ run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)wait_till_workers_are_ready (data);rte_delay_ms (100 );/* Wait for some time to accumulate the stats */ -measure_perf (str, data);signal_workers_to_finish (data);RTE_LCORE_FOREACH_WORKER (id)rte_eal_wait_lcore (id); + +measure_perf (str, data); } static int
Signature · 1 line
2.17.1
v3 cover remove smp barriers in app/test
In reply to Feifei WangFor smp barriers in app/test, remove the unnecessary barriers and fix some bugs.
v2:
1. remove ring stress test patch for that Honnappa will do some changes for rte_remote_launch APIs
v3:
1. change the patch tag (Jerin)
Feifei Wang (1):
test/trace: collect perf data after worker threads exit
app/test/test_trace_perf.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
Signature · 1 line
2.25.1
PATCH v3 1/1 test/trace: collect perf data after worker threads exit
The measure_perf function should be executed after worker threads exit to collect correct perf data. Otherwise, while workers are running, the main thread may get incomplete data from workers.
In the meanwhile, remove unnecessary barrier in the test. For signal variables "ldata.done" and "ldata.start", no operations should keep the order that being executed after them. So the wmb after them can be moved.
--- app/test/test_trace_perf.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
app/test/test_trace_perf.c +2 -3
@@ -79,7 +79,6 @@ signal_workers_to_finish(struct test_data *data)for (workers =0 ; workers < data->nb_workers; workers++) { data->ldata[workers].done =1 ; -rte_smp_wmb (); } }
@@ -102,7 +101,6 @@ worker_fn_##func(void *arg) \ { \struct lcore_data *ldata = arg; \ ldata->started =1 ; \ -rte_smp_wmb (); \ __worker_##func(ldata); \ return 0; \ }
@@ -137,11 +135,12 @@ run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)wait_till_workers_are_ready (data);rte_delay_ms (100 );/* Wait for some time to accumulate the stats */ -measure_perf (str, data);signal_workers_to_finish (data);RTE_LCORE_FOREACH_WORKER (id)rte_eal_wait_lcore (id); + +measure_perf (str, data); } static int
Signature · 1 line
2.25.1
v3 cover remove smp barriers in app/test
In reply to Feifei WangFor smp barriers in app/test, remove the unnecessary barriers and fix some bugs.
v2:
1. remove ring stress test patch for that Honnappa will do some changes for rte_remote_launch APIs
v3:
1. change the patch tag (Jerin)
Feifei Wang (1):
test/trace: collect perf data after worker threads exit
app/test/test_trace_perf.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
Signature · 1 line
2.25.1
PATCH v3 1/1 test/trace: collect perf data after worker threads exit
The measure_perf function should be executed after worker threads exit to collect correct perf data. Otherwise, while workers are running, the main thread may get incomplete data from workers.
In the meanwhile, remove unnecessary barrier in the test. For signal variables "ldata.done" and "ldata.start", no operations should keep the order that being executed after them. So the wmb after them can be moved.
--- app/test/test_trace_perf.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
app/test/test_trace_perf.c +2 -3
@@ -79,7 +79,6 @@ signal_workers_to_finish(struct test_data *data)for (workers =0 ; workers < data->nb_workers; workers++) { data->ldata[workers].done =1 ; -rte_smp_wmb (); } }
@@ -102,7 +101,6 @@ worker_fn_##func(void *arg) \ { \struct lcore_data *ldata = arg; \ ldata->started =1 ; \ -rte_smp_wmb (); \ __worker_##func(ldata); \ return 0; \ }
@@ -137,11 +135,12 @@ run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)wait_till_workers_are_ready (data);rte_delay_ms (100 );/* Wait for some time to accumulate the stats */ -measure_perf (str, data);signal_workers_to_finish (data);RTE_LCORE_FOREACH_WORKER (id)rte_eal_wait_lcore (id); + +measure_perf (str, data); } static int
Signature · 1 line
2.25.1
David Marchand
AppliedRe: [dpdk-dev] [PATCH v3 1/1] test/trace: collect perf data after worker threads exit
Quoted 13 lines · Feifei Wang
The
measure_perffunction should be executed after worker threads exit to collect correct perf data. Otherwise, while workers are running, the main thread may get incomplete data from workers.In the meanwhile, remove unnecessary barrier in the test. For signal variables "ldata.done" and "ldata.start", no operations should keep the order that being executed after them. So the wmb after them can be moved.
Fixes 16a277a24c9f ("test/trace: add performance test cases")
Quoted 6 lines
Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Feifei Wang <[email protected]>Reviewed-by Honnappa Nagarahalli <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Acked-by Pavan Nikhilesh <[email protected]>Acked-by Jerin Jacob <[email protected]>
Applied, thanks.
Signature · 1 line
David Marchand