Skip to contentSkip to conversation

remove smp barriers in app/test

PATCH Superseded Feifei Wang 5 participants 19 messages Dec 22, 2020 – Apr 14, 2021

lore
Share & tools
lore
#0 lore raw

cover remove smp barriers in app/test

For smp barriers in app/test, remove the unnecessary barriers and fix some bugs.

Feifei Wang (2):
app/test: remove unnecessary barriers for ring stress test app/test: collect perf data after worker threads exit

 app/test/test_ring_stress_impl.h | 5 +----
 app/test/test_trace_perf.c       | 5 ++---
 2 files changed, 3 insertions(+), 7 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/2 app/test: remove unnecessary barriers for ring stress test

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd == WRK_CMD_RUN", they start running and when worker lcores load "wrk_cmd == WRK_CMD_STOP", they stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

For the rmb in test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to wait wrk_cmd to be loaded, then the parameters can be loaded, So the rmb can be removed.

In the meanwhile, fix a typo. The note above storing "stop" into "wrk_cmd" should be "stop test" rather than "start test".

---
 app/test/test_ring_stress_impl.h | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
app/test/test_ring_stress_impl.h +1 -4
index f9ca63b90..384555ef9 100644
@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname, int32_t prcs)
 	fill_ring_elm(&loc_elm, lc);
 
 	while (wrk_cmd != WRK_CMD_RUN) {
-		rte_smp_rmb();
 		rte_pause();
 	}
 
@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))
 
 	/* signal worker to start test */
 	wrk_cmd = WRK_CMD_RUN;
-	rte_smp_wmb();
 
 	usleep(run_time * US_PER_S);
 
-	/* signal worker to start test */
+	/* signal worker to stop test */
 	wrk_cmd = WRK_CMD_STOP;
-	rte_smp_wmb();
 
 	/* wait for workers and collect stats. */
 	mc = rte_lcore_id();
Signature · 1 line
2.17.1
#2 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

Hi Feifei,

Quoted 8 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd == WRK_CMD_RUN", they start running and when worker lcores load "wrk_cmd == WRK_CMD_STOP", they stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb() *before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;

- rte_smp_wmb();

        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd = WRK_CMD_RUN;' But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.

Quoted 3 lines

For the rmb in test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to wait wrk_cmd to be loaded, then the parameters can be loaded, So the rmb can be removed.

It is not only about parameters loading, it is to prevent worker core to start too early.

As I understand, your goal is to get rid of rte_smp_*() calls. Might be better to replace such places here with _atomic_ semantics. Then, as I can see, we also can get rid of 'volatile' fo wrk_cmd.

Quoted 39 lines

In the meanwhile, fix a typo. The note above storing "stop" into "wrk_cmd" should be "stop test" rather than "start test".

---
 app/test/test_ring_stress_impl.h | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
app/test/test_ring_stress_impl.h +1 -5
index f9ca63b90..384555ef9 100644
@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname, int32_t prcs)
 	fill_ring_elm(&loc_elm, lc);
 
 	while (wrk_cmd != WRK_CMD_RUN) {
-		rte_smp_rmb();
 		rte_pause();
 	}
 
@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))
 
 	/* signal worker to start test */
 	wrk_cmd = WRK_CMD_RUN;
-	rte_smp_wmb();
 
 	usleep(run_time * US_PER_S);
 
-	/* signal worker to start test */
+	/* signal worker to stop test */
 	wrk_cmd = WRK_CMD_STOP;
-	rte_smp_wmb();
 
 	/* wait for workers and collect stats. */
 	mc = rte_lcore_id();
--

2.17.1

#3 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

<snip>

Quoted 32 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd == WRK_CMD_RUN", they

start

Quoted 1 line

running and when worker lcores load "wrk_cmd == WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb() *before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
app/test/test_ring_stress_impl.h +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
 
        usleep(run_time * US_PER_S);
Quoted 7 lines

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd = WRK_CMD_RUN;' But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new thread. The rte_smp_rmb() barrier in the thread function is not required as it reads the data that was set before the thread was launched.

I do not know why rte_smp_wmb is required. The update to 'wrk_cmd' is seen by the thread eventually. rte_smp_wmb does not result in update being seen sooner/immediately.

[1] https://www.youtube.com/watch?t=4170&v=KeLBd2EJLOU&feature=youtu.be

Quoted 8 lines
Quoted 4 lines

For the rmb in test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to wait wrk_cmd to be loaded, then the parameters can be loaded, So the rmb can be

removed.

It is not only about parameters loading, it is to prevent worker core to start too early.

Because 'pthread_launch' provides the 'happens-before' behavior, the worker core will see the updates that happened before the worker was launched.

I suggest changing the commit log to provide the reasoning around pthread_create.

Quoted 46 lines

As I understand, your goal is to get rid of rte_smp_*() calls. Might be better to replace such places here with _atomic_ semantics. Then, as I can see, we also can get rid of 'volatile' fo wrk_cmd.

app/test/test_ring_stress_impl.h test_worker(void *arg, const char *fname, int32_t

Quoted 16 lines

In the meanwhile, fix a typo. The note above storing "stop" into "wrk_cmd" should be "stop test" rather than "start test".

---
 app/test/test_ring_stress_impl.h | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
app/test/test_ring_stress_impl.h +0 -0
index f9ca63b90..384555ef9 100644
@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname, int32_t

prcs)

Quoted 24 lines
app/test/test_ring_stress_impl.h +1 -5
 	fill_ring_elm(&loc_elm, lc);
 
 	while (wrk_cmd != WRK_CMD_RUN) {
-		rte_smp_rmb();
 		rte_pause();
 	}
 
@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))
 
 	/* signal worker to start test */
 	wrk_cmd = WRK_CMD_RUN;
-	rte_smp_wmb();
 
 	usleep(run_time * US_PER_S);
 
-	/* signal worker to start test */
+	/* signal worker to stop test */
 	wrk_cmd = WRK_CMD_STOP;
-	rte_smp_wmb();
 
 	/* wait for workers and collect stats. */
 	mc = rte_lcore_id();
--

2.17.1

#4 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

Quoted 40 lines
Quoted 33 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd == WRK_CMD_RUN", they

start

Quoted 1 line

running and when worker lcores load "wrk_cmd == WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb() *before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
app/test/test_ring_stress_impl.h +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
 
        usleep(run_time * US_PER_S);
Quoted 7 lines

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd = WRK_CMD_RUN;' But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.

Quoted 5 lines

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new thread. The rte_smp_rmb() barrier in the thread function is not required as it reads the data that was set before the thread was launched.

rte_eal_remote_launch() doesn't call pthread_create().
All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Quoted 3 lines

I do not know why rte_smp_wmb is required. The update to 'wrk_cmd' is seen by the thread eventually. rte_smp_wmb does not result in update being seen sooner/immediately.

We don't need it sooner.
We need to make sure it wouldn't be seen by any worker thread before all workers are launched. To make sure all workers start the test at approximately same moment.
That's why I think wmb() should be before 'wrk_cmd = WRK_CMD_RUN;' in my original code.

Quoted 61 lines

[1] https://www.youtube.com/watch?t=4170&v=KeLBd2EJLOU&feature=youtu.be

Quoted 8 lines
Quoted 4 lines

For the rmb in test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to wait wrk_cmd to be loaded, then the parameters can be loaded, So the rmb can be

removed.

It is not only about parameters loading, it is to prevent worker core to start too early.

Because 'pthread_launch' provides the 'happens-before' behavior, the worker core will see the updates that happened before the worker was launched.

I suggest changing the commit log to provide the reasoning around pthread_create.

Quoted 46 lines

As I understand, your goal is to get rid of rte_smp_*() calls. Might be better to replace such places here with _atomic_ semantics. Then, as I can see, we also can get rid of 'volatile' fo wrk_cmd.

app/test/test_ring_stress_impl.h test_worker(void *arg, const char *fname, int32_t

Quoted 16 lines

In the meanwhile, fix a typo. The note above storing "stop" into "wrk_cmd" should be "stop test" rather than "start test".

---
 app/test/test_ring_stress_impl.h | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
app/test/test_ring_stress_impl.h +0 -0
index f9ca63b90..384555ef9 100644
@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname, int32_t

prcs)

Quoted 24 lines
app/test/test_ring_stress_impl.h +1 -5
 	fill_ring_elm(&loc_elm, lc);
 
 	while (wrk_cmd != WRK_CMD_RUN) {
-		rte_smp_rmb();
 		rte_pause();
 	}
 
@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))
 
 	/* signal worker to start test */
 	wrk_cmd = WRK_CMD_RUN;
-	rte_smp_wmb();
 
 	usleep(run_time * US_PER_S);
 
-	/* signal worker to start test */
+	/* signal worker to stop test */
 	wrk_cmd = WRK_CMD_STOP;
-	rte_smp_wmb();
 
 	/* wait for workers and collect stats. */
 	mc = rte_lcore_id();
--

2.17.1

#5 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

<snip>

Quoted 51 lines
Quoted 7 lines
Quoted 7 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd ==

WRK_CMD_RUN",

Quoted 24 lines
Quoted 24 lines
Quoted 1 line

they

start

Quoted 1 line

running and when worker lcores load "wrk_cmd == WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb() *before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
app/test/test_ring_stress_impl.h +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
Quoted 8 lines
Quoted 8 lines
        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd =

WRK_CMD_RUN;'

Quoted 10 lines
Quoted 2 lines

But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new thread. The
rte_smp_rmb() barrier in the thread function is not required as it reads the

Quoted 6 lines

data that was set before the thread was launched.

rte_eal_remote_launch() doesn't call pthread_create().
All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide behavior similar to pthread_launch and pthread_join respectively.

There is use of rte_smp_*mb in those functions as well. Those need to be fixed first and then look at these.

Quoted 80 lines
Quoted 3 lines

I do not know why rte_smp_wmb is required. The update to 'wrk_cmd' is seen by the thread eventually. rte_smp_wmb does not result in update

being seen sooner/immediately.

We don't need it sooner.
We need to make sure it wouldn't be seen by any worker thread before all workers are launched.
To make sure all workers start the test at approximately same moment. That's why I think wmb() should be before 'wrk_cmd = WRK_CMD_RUN;' in my original code.

Quoted 3 lines

[1]

https://www.youtube.com/watch?t=4170&v=KeLBd2EJLOU&feature=youtu.
be

Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

For the rmb in test_worker, the parameters have been prepared when worker lcores call "test_worker". It is unnessary to wait wrk_cmd to be loaded, then the parameters can be loaded, So the rmb can be

removed.

It is not only about parameters loading, it is to prevent worker core to start too early.

Because 'pthread_launch' provides the 'happens-before' behavior, the worker core will see the updates that happened before the worker was

launched.

Quoted 2 lines

I suggest changing the commit log to provide the reasoning around

pthread_create.

Quoted 11 lines
Quoted 11 lines

As I understand, your goal is to get rid of rte_smp_*() calls. Might be better to replace such places here with _atomic_ semantics. Then, as I can see, we also can get rid of 'volatile' fo wrk_cmd.

Quoted 5 lines

In the meanwhile, fix a typo. The note above storing "stop" into "wrk_cmd" should be "stop test" rather than "start test".

Reviewed-by Honnappa Nagarahalli

<[email protected]>

Quoted 37 lines
Quoted 37 lines
Quoted 11 lines
---
 app/test/test_ring_stress_impl.h | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
app/test/test_ring_stress_impl.h +0 -0
index f9ca63b90..384555ef9 100644
@@ -198,7 +198,6 @@ test_worker(void *arg, const char *fname,
Quoted 1 line

int32_t

prcs)

Quoted 24 lines
app/test/test_ring_stress_impl.h +1 -5
 	fill_ring_elm(&loc_elm, lc);
 
 	while (wrk_cmd != WRK_CMD_RUN) {
-		rte_smp_rmb();
 		rte_pause();
 	}
@@ -357,13 +356,11 @@ test_mt1(int (*test)(void *))
 
 	/* signal worker to start test */
 	wrk_cmd = WRK_CMD_RUN;
-	rte_smp_wmb();
 
 	usleep(run_time * US_PER_S);
 
-	/* signal worker to start test */
+	/* signal worker to stop test */
 	wrk_cmd = WRK_CMD_STOP;
-	rte_smp_wmb();
 
 	/* wait for workers and collect stats. */
 	mc = rte_lcore_id();
--

2.17.1

#6 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

Quoted 55 lines · On Fri, 29 Jan 2021 03:17:50 +0000 Honnappa Nagarahalli…

<snip>

Quoted 53 lines
Quoted 7 lines
Quoted 7 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd ==

WRK_CMD_RUN",

Quoted 24 lines
Quoted 24 lines
Quoted 1 line

they

start

Quoted 1 line

running and when worker lcores load "wrk_cmd == WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb() *before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
app/test/test_ring_stress_impl.h +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
Quoted 8 lines
Quoted 8 lines
        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)?
We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd =

WRK_CMD_RUN;'

Quoted 10 lines
Quoted 2 lines

But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical pass.

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new thread. The
rte_smp_rmb() barrier in the thread function is not required as it reads the

data that was set before the thread was launched.

Quoted 8 lines
Quoted 4 lines

rte_eal_remote_launch() doesn't call pthread_create().
All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide behavior similar to pthread_launch and pthread_join respectively.

There is use of rte_smp_*mb in those functions as well. Those need to be fixed first and then look at these.

Looks like you want __atomic_thread_fence() here.

#7 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

<snip>

Quoted 63 lines
Quoted 43 lines
Quoted 43 lines
Quoted 8 lines
Quoted 8 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd ==

WRK_CMD_RUN",

Quoted 34 lines
Quoted 34 lines
Quoted 1 line

they

start

Quoted 1 line

running and when worker lcores load "wrk_cmd == WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb()
*before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
app/test/test_ring_stress_impl.h +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
 
        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd =

Quoted 3 lines
Quoted 3 lines

WRK_CMD_RUN;'

Quoted 2 lines
Quoted 2 lines

But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical

pass.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new

thread.

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

The
rte_smp_rmb() barrier in the thread function is not required as it reads the

data that was set before the thread was launched.

rte_eal_remote_launch() doesn't call pthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Quoted 9 lines
Quoted 2 lines

Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide

behavior similar to pthread_launch and pthread_join respectively.

Quoted 2 lines

There is use of rte_smp_*mb in those functions as well. Those need to be fixed

first and then look at these.

Looks like you want __atomic_thread_fence() here.

In the rte_eal_remote_launch case, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In the eal_thread_loop function we could do load-acquire on the function pointer.

I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).

The rte_eal_wait_lcore API needs to ensure that the memory operations in the worker are visible to the main. rte_eal_wait_lcore and eal_thread_loop are synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' in eal_thread_loop and a load-acquire in rte_eal_wait_lcore.

So, we do not have to use the __atomic_thread_fence.

#8 lore raw

[dpdk-dev] 回复: [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

Hi, Honnappa, Konstantin and Stephen

Thanks very much for your attention of this patch. Based on your opinion, Ruifeng and I discuss about this and make a summary:
____________________________________________________________________________________________________________________
main thread worker thread
rte_eal_remote_launch: [ Honnappa focus ]

																		To ensure f can load correct arg,
																		arg store should before f
						lcore_config[worker_id].f = f;
						lcore_config[worker_id].arg = arg;
						wmb()? or store-relase on f?
						
										eal_thread_loop:
						pipeline_communication	---------------------->	pipeline_communication
													if (lcore_config[lcore_id].f == NULL)
													rte_panic("NULL function pointer\n");
		
													fct_arg = lcore_config[lcore_id].arg;
													ret = lcore_config[lcore_id].f(fct_arg);

___________________________________________________________________________________________________________________

test_ring_stress: wmb()? [ Konstantin focus ]

										test_worker:							Main thread can use wrk_cmd to
						Wrk_cmd =WRK_CMD_RUN;	---------------------->	Wrk_cmd == WRK_CMD_RUN;		control multiple threads to start running
													wmb()?					at the same time as much as possible
													ring_dequeue;
													ring_enqueue;
						Wrk_cmd =WRK_CMD_STOP;	---------------------->	Wrk_cmd == WRK_CMD_STOP;

____________________________________________________________________________________________________________________

rte_eal_wait_lcore: wmb() [ Honnappa focus ]

				lcore_config[lcore_id].state == FINISHED	<---------------------	lcore_config[lcore_id].state = FINISHED	Load-acquire and store-release
																		are used on the variable “state”
						rmb();

____________________________________________________________________________________________________________________

From the picture above,

1.First, for the underlying function rte_eal_remote_launch, Honnappa focuses on that, pipeline_communication cannot ensure ‘arg’ parameters is loaded correctly by the worker thread.
This is because in weak memory order framework, maybe the main thread and worker thread firstly finish pipeline communication, and then the worker thread receive signal and execute the function ‘ f ’. However, it maybe load a wrong value of ‘arg’ due to that the main thread stores ‘arg’ after pipeline communication. So wmb or store_release is necessary for ‘arg’.

2.Second, for the upper-layer test_ring_stress, Konstantin foucese on that, Whether the main thread can use ‘wrk_cmd’ to control multiple threads to run at the same time as much as possible.
Because rte_eal_remote_launch only can communicates with one worker thread
at the same time. This means some worker thread maybe start working very early but other worker threads maybe need to wait a long time to start working if ‘wrk_cmd' is stored 'RUN' flag before rte_remote_launch.
At last, for unit test, this may cause that the test results are not stable.

3.Third, for rte_eal_wait_lcore, Honnappa focuses on that the ‘state’ as a synchronous bariable,
we should add load-acquire and store-release on it. However, there have been rmb and wmb after and before ‘state’, So I’m not sure whether we should replace them.

In summary, I think Honnappa and Konstantin have different concerns.
For Honnappa, we can add wmb or store-release to ensure the ‘arg’ can be loaded correctly in rte_eal_remote_launch.
For Konstantin, we can add wmb and rmb to ensure the main thread can control the worker Threads to run at the same time, and then make the test results more accurate in the ring_stress_test.

Best Regards
Feifei

Quoted 109 lines

-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

<snip>

Quoted 68 lines
Quoted 44 lines
Quoted 44 lines
Quoted 8 lines
Quoted 8 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd ==

WRK_CMD_RUN",

Quoted 35 lines
Quoted 35 lines
Quoted 1 line

they

start

Quoted 2 lines

running and when worker lcores load "wrk_cmd ==
WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb()
*before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
(quoted diff) +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
 
        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd =

Quoted 4 lines
Quoted 4 lines

WRK_CMD_RUN;'

Quoted 3 lines
Quoted 3 lines

But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical

pass.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new

thread.

Quoted 11 lines
Quoted 9 lines
Quoted 3 lines

The
rte_smp_rmb() barrier in the thread function is not required as it reads the

data that was set before the thread was launched.

rte_eal_remote_launch() doesn't call pthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide

Quoted 8 lines

behavior similar to pthread_launch and pthread_join respectively.

Quoted 3 lines

There is use of rte_smp_*mb in those functions as well. Those need to be fixed

first and then look at these.

Looks like you want __atomic_thread_fence() here.

In the rte_eal_remote_launch case, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In the eal_thread_loop function we could do load-acquire on the function pointer.

I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).

The rte_eal_wait_lcore API needs to ensure that the memory operations in the worker are visible to the main. rte_eal_wait_lcore and eal_thread_loop are synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' in eal_thread_loop and a load-acquire in rte_eal_wait_lcore.

So, we do not have to use the __atomic_thread_fence.

#9 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

Hi Feifei,

Quoted 79 lines

Hi, Honnappa, Konstantin and Stephen

Thanks very much for your attention of this patch. Based on your opinion, Ruifeng and I discuss about this and make a summary:
_________________________________________________________________________________________________________________
___
main thread worker thread
rte_eal_remote_launch:

	[ Honnappa focus ]

	To ensure f can load correct arg,

	arg store should before f
						lcore_config[worker_id].f = f;
						lcore_config[worker_id].arg = arg;
						wmb()? or store-relase on f?

										eal_thread_loop:
						pipeline_communication	---------------------->	pipeline_communication
													if (lcore_config[lcore_id].f ==

NULL)
rte_panic("NULL function
pointer\n");

fct_arg =
lcore_config[lcore_id].arg;
ret =
lcore_config[lcore_id].f(fct_arg);
_________________________________________________________________________________________________________________
__

test_ring_stress: wmb()?

	[ Konstantin focus ]
										test_worker:
	Main thread can use wrk_cmd to
						Wrk_cmd =WRK_CMD_RUN;	---------------------->	Wrk_cmd == WRK_CMD_RUN;
	control multiple threads to start running
													wmb()?
	at the same time as much as possible
													ring_dequeue;
													ring_enqueue;
						Wrk_cmd =WRK_CMD_STOP;	---------------------->	Wrk_cmd == WRK_CMD_STOP;

_________________________________________________________________________________________________________________
___

rte_eal_wait_lcore: wmb()

	[ Honnappa focus ]
				lcore_config[lcore_id].state == FINISHED	<---------------------	lcore_config[lcore_id].state =

FINISHED Load-acquire and store-release

	are used on the variable “state”
						rmb();

_________________________________________________________________________________________________________________
___

From the picture above,

1.First, for the underlying function rte_eal_remote_launch, Honnappa focuses on that, pipeline_communication cannot ensure ‘arg’ parameters is loaded correctly by the worker thread.
This is because in weak memory order framework, maybe the main thread and worker thread firstly finish pipeline communication, and then the worker thread receive signal and execute the function ‘ f ’. However, it maybe load a wrong value of ‘arg’ due to that the main thread stores ‘arg’ after pipeline communication. So wmb or store_release is necessary for ‘arg’.

2.Second, for the upper-layer test_ring_stress, Konstantin foucese on that, Whether the main thread can use ‘wrk_cmd’ to control multiple threads to run at the same time as much as possible.
Because rte_eal_remote_launch only can communicates with one worker thread
at the same time. This means some worker thread maybe start working very early but other worker threads maybe need to wait a long time to start working if ‘wrk_cmd' is stored 'RUN' flag before rte_remote_launch.
At last, for unit test, this may cause that the test results are not stable.

3.Third, for rte_eal_wait_lcore, Honnappa focuses on that the ‘state’ as a synchronous bariable,
we should add load-acquire and store-release on it. However, there have been rmb and wmb after and before ‘state’, So I’m not sure whether we should replace them.

Quoted 6 lines

In summary, I think Honnappa and Konstantin have different concerns.
For Honnappa, we can add wmb or store-release to ensure the ‘arg’ can be loaded correctly in rte_eal_remote_launch.
For Konstantin, we can add wmb and rmb to ensure the main thread can control the worker Threads to run at the same time, and then make the test results more accurate in the ring_stress_test.

Agree with both.

Quoted 114 lines

Best Regards
Feifei

Quoted 109 lines

-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

<snip>

Quoted 68 lines
Quoted 44 lines
Quoted 44 lines
Quoted 8 lines
Quoted 8 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd ==

WRK_CMD_RUN",

Quoted 35 lines
Quoted 35 lines
Quoted 1 line

they

start

Quoted 2 lines

running and when worker lcores load "wrk_cmd ==
WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb()
*before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
(quoted diff) +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
 
        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd =

Quoted 4 lines
Quoted 4 lines

WRK_CMD_RUN;'

Quoted 3 lines
Quoted 3 lines

But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical

pass.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new

thread.

Quoted 11 lines
Quoted 9 lines
Quoted 3 lines

The
rte_smp_rmb() barrier in the thread function is not required as it reads the

data that was set before the thread was launched.

rte_eal_remote_launch() doesn't call pthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide

Quoted 8 lines

behavior similar to pthread_launch and pthread_join respectively.

Quoted 3 lines

There is use of rte_smp_*mb in those functions as well. Those need to be fixed

first and then look at these.

Looks like you want __atomic_thread_fence() here.

In the rte_eal_remote_launch case, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In the eal_thread_loop function we could do load-acquire on the function pointer.

I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).

The rte_eal_wait_lcore API needs to ensure that the memory operations in the worker are visible to the main. rte_eal_wait_lcore and eal_thread_loop are synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' in eal_thread_loop and a load-acquire in rte_eal_wait_lcore.

So, we do not have to use the __atomic_thread_fence.

#10 lore raw

Re: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

<snip>

Quoted 115 lines

Hi Feifei,

Quoted 6 lines

Hi, Honnappa, Konstantin and Stephen

Thanks very much for your attention of this patch. Based on your opinion, Ruifeng and I discuss about this and make a summary:

__________________________________________________________
____________

Quoted 3 lines

___________________________________________
___
main thread

worker thread

Quoted 8 lines

rte_eal_remote_launch:
[ Honnappa focus ]

To ensure f can load correct arg,

arg store should before f
lcore_config[worker_id].f = f; lcore_config[worker_id].arg =

arg;

Quoted 3 lines

wmb()? or store-relase on f?

eal_thread_loop:

Quoted 1 line

pipeline_communication

	---------------------->	pipeline_communication
Quoted 1 line
				if (lcore_config[lcore_id].f ==
Quoted 2 lines

NULL)

rte_panic("NULL function

Quoted 3 lines

pointer\n");

fct_arg =

Quoted 2 lines

lcore_config[lcore_id].arg;

ret =

Quoted 2 lines

lcore_config[lcore_id].f(fct_arg);

__________________________________________________________
____________

Quoted 6 lines

___________________________________________
__

test_ring_stress: wmb()?
[ Konstantin focus ]

test_worker:

Quoted 2 lines
	Main thread can use wrk_cmd to
						Wrk_cmd =WRK_CMD_RUN;
	---------------------->	Wrk_cmd == WRK_CMD_RUN;
Quoted 2 lines

control multiple threads to start running

				wmb()?
Quoted 2 lines

at the same time as much as possible

				ring_dequeue;
Quoted 1 line
				ring_enqueue;
Quoted 1 line
						Wrk_cmd =WRK_CMD_STOP;
	---------------------->	Wrk_cmd == WRK_CMD_STOP;
Quoted 1 line

__________________________________________________________
____________

Quoted 4 lines

___________________________________________
___

rte_eal_wait_lcore:

				wmb()
Quoted 2 lines
	[ Honnappa focus ]
				lcore_config[lcore_id].state == FINISHED
	<---------------------	lcore_config[lcore_id].state =
Quoted 5 lines

FINISHED Load-acquire and store-release

	are used on the variable “state”
						rmb();

__________________________________________________________
____________

Quoted 5 lines

___________________________________________
___

From the picture above,

Quoted 8 lines

1.First, for the underlying function rte_eal_remote_launch, Honnappa focuses on that, pipeline_communication cannot ensure ‘arg’ parameters is loaded correctly by the worker thread.
This is because in weak memory order framework, maybe the main thread and worker thread firstly finish pipeline communication, and then the worker thread receive signal and execute the function ‘ f ’. However, it maybe load a wrong value of ‘arg’ due to that the main thread stores ‘arg’ after pipeline communication. So wmb or store_release is

necessary for ‘arg’.

Quoted 8 lines

2.Second, for the upper-layer test_ring_stress, Konstantin foucese on that, Whether the main thread can use ‘wrk_cmd’ to control multiple threads to run at the same time as much as possible. Because rte_eal_remote_launch only can communicates with one worker thread at the same time. This means some worker thread maybe start working very early but other worker threads maybe need to wait a long time to start working if ‘wrk_cmd' is stored 'RUN' flag before

rte_remote_launch.

Quoted 3 lines

At last, for unit test, this may cause that the test results are not stable.

3.Third, for rte_eal_wait_lcore, Honnappa focuses on that the ‘state’ as a

synchronous bariable,

Quoted 2 lines

we should add load-acquire and store-release on it. However, there have been rmb and wmb after and before ‘state’, So I’m not sure whether

we should replace them.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

In summary, I think Honnappa and Konstantin have different concerns. For Honnappa, we can add wmb or store-release to ensure the ‘arg’ can be loaded correctly in rte_eal_remote_launch.
For Konstantin, we can add wmb and rmb to ensure the main thread can control the worker Threads to run at the same time, and then make the test results more accurate in the ring_stress_test.

Agree with both.

Thanks Feifei, understood. I am just trying to take a step back and see what kind of ordering guarantees rte_eal_remote_launch should provide so that we do not have to deal with adding additional barriers in the applications. For ex: if we can avoid the barriers around 'wrk_cmd' (kind of use cases) it will benefit all the applications.

Quoted 121 lines
Quoted 11 lines

Best Regards
Feifei

Quoted 5 lines

-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]> 抄送: Ananyev, Konstantin <[email protected]>; Feifei

Wang

Quoted 1 line
Quoted 1 line

<[email protected]>; [email protected]; nd <[email protected]>; Ruifeng

Wang

Quoted 18 lines
Quoted 18 lines

<[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

<snip>

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

Hi Feifei,

Quoted 4 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd ==

WRK_CMD_RUN",

Quoted 1 line
Quoted 1 line
Quoted 1 line

they

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

start

Quoted 2 lines

running and when worker lcores load "wrk_cmd ==
WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is

unnecessary.

Quoted 71 lines
Quoted 71 lines
Quoted 51 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

I think there is a bug in my original code, we should do smp_wmb()
*before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
(quoted diff) +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
 
        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd =

Quoted 4 lines
Quoted 4 lines

WRK_CMD_RUN;'

Quoted 3 lines
Quoted 3 lines

But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical

pass.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new

thread.

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

The
rte_smp_rmb() barrier in the thread function is not required as it reads the

data that was set before the thread was launched.

Quoted 7 lines
Quoted 4 lines

rte_eal_remote_launch() doesn't call pthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide

Quoted 8 lines

behavior similar to pthread_launch and pthread_join respectively.

Quoted 3 lines

There is use of rte_smp_*mb in those functions as well. Those need to be fixed

first and then look at these.

Looks like you want __atomic_thread_fence() here.

In the rte_eal_remote_launch case, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In the eal_thread_loop function we could do load-acquire on the function pointer.

I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).

Quoted 5 lines
Quoted 5 lines

The rte_eal_wait_lcore API needs to ensure that the memory operations in the worker are visible to the main. rte_eal_wait_lcore and eal_thread_loop are synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on

'state' in eal_thread_loop and a load-acquire in rte_eal_wait_lcore.

Quoted 3 lines
Quoted 3 lines

So, we do not have to use the __atomic_thread_fence.

#11 lore raw

[dpdk-dev] 回复: [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

In reply to Honnappa Nagarahalli

Sorry, a mistake happens in the picture, after Wrk_cmd == WRK_CMD_RUN, it should be a rmb rather than wmb.

Quoted 109 lines

-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

<snip>

Quoted 68 lines
Quoted 44 lines
Quoted 44 lines
Quoted 8 lines
Quoted 8 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd ==

WRK_CMD_RUN",

Quoted 35 lines
Quoted 35 lines
Quoted 1 line

they

start

Quoted 2 lines

running and when worker lcores load "wrk_cmd ==
WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is unnecessary.

I think there is a bug in my original code, we should do smp_wmb()
*before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
(quoted diff) +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
 
        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd =

Quoted 4 lines
Quoted 4 lines

WRK_CMD_RUN;'

Quoted 3 lines
Quoted 3 lines

But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical

pass.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new

thread.

Quoted 11 lines
Quoted 9 lines
Quoted 3 lines

The
rte_smp_rmb() barrier in the thread function is not required as it reads the

data that was set before the thread was launched.

rte_eal_remote_launch() doesn't call pthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide

Quoted 8 lines

behavior similar to pthread_launch and pthread_join respectively.

Quoted 3 lines

There is use of rte_smp_*mb in those functions as well. Those need to be fixed

first and then look at these.

Looks like you want __atomic_thread_fence() here.

In the rte_eal_remote_launch case, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In the eal_thread_loop function we could do load-acquire on the function pointer.

I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).

The rte_eal_wait_lcore API needs to ensure that the memory operations in the worker are visible to the main. rte_eal_wait_lcore and eal_thread_loop are synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' in eal_thread_loop and a load-acquire in rte_eal_wait_lcore.

So, we do not have to use the __atomic_thread_fence.

#12 lore raw

[dpdk-dev] 回复: [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

Hi, everyone

Sorry for that there may be a problem in the e-mail format. Please see the picture according to the following link:
https://ibb.co/SQ7yGfW

Best Regards
Feifei

Quoted 127 lines

-----邮件原件-----
发件人: Feifei Wang
发送时间: 2021年2月1日 16:49
收件人: Honnappa Nagarahalli <[email protected]>; Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng Wang <[email protected]>; nd <[email protected]>
主题: 回复: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

Sorry, a mistake happens in the picture, after Wrk_cmd == WRK_CMD_RUN, it should be a rmb rather than wmb.

Quoted 6 lines

-----邮件原件-----
发件人: Honnappa Nagarahalli <[email protected]> 发送时间: 2021年1月30日 9:24
收件人: Stephen Hemminger <[email protected]>
抄送: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; [email protected]; nd <[email protected]>; Ruifeng

Wang

Quoted 5 lines

<[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
主题: RE: [dpdk-dev] [PATCH v1 1/2] app/test: remove unnecessary barriers for ring stress test

Quoted 20 lines

<snip>

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines

Hi Feifei,

Quoted 3 lines

The variable "wrk_cmd" is a signal to control threads from running and stopping. When worker lcores load "wrk_cmd ==

WRK_CMD_RUN",

Quoted 9 lines
Quoted 9 lines
Quoted 1 line

they

start

Quoted 2 lines

running and when worker lcores load "wrk_cmd ==
WRK_CMD_STOP",

they

Quoted 4 lines

stop.

For the wmb in test_mt1, no storing operations must keep the order after storing "wrk_cmd". Thus the wmb is

unnecessary.

Quoted 71 lines
Quoted 51 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

I think there is a bug in my original code, we should do smp_wmb()
*before* setting wrk_cmd, not after:

        /* launch on all workers */
        RTE_LCORE_FOREACH_WORKER(lc) {
                arg[lc].rng = r;
                arg[lc].stats = init_stat;
                rte_eal_remote_launch(test, &arg[lc], lc);
        }

        /* signal worker to start test */
(quoted diff) +1 -1
+      rte_smp_wmb();
        wrk_cmd = WRK_CMD_RUN;
-       rte_smp_wmb();
 
        usleep(run_time * US_PER_S);

I still think we'd better have some synchronisation here. Otherwise what would prevent compiler and/or cpu to update wrk_cmd out of order (before _init_ phase is completed)? We probably can safely assume no reordering from the compiler here, as we have function calls straight before and after 'wrk_cmd =

Quoted 4 lines
Quoted 4 lines

WRK_CMD_RUN;'

Quoted 3 lines
Quoted 3 lines

But for consistency and easier maintenance, I still think it is better to have something here, after all it is not performance critical

pass.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Agree that this is not performance critical.

This is more about correctness (as usually people refer to code to understand the concepts). You can refer to video [1]. Essentially, the pthread_create has 'happens-before' behavior. i.e. all the memory operations before the pthread_create are visible to the new

thread.

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

The
rte_smp_rmb() barrier in the thread function is not required as it reads the

data that was set before the thread was launched.

Quoted 7 lines
Quoted 4 lines

rte_eal_remote_launch() doesn't call pthread_create(). All it does - updates global variable (lcore_config) and writes/reads to/from the pipe.

Thanks for the reminder ☹
I think rte_eal_remote_launch and rte_eal_wait_lcore need to provide

Quoted 8 lines

behavior similar to pthread_launch and pthread_join respectively.

Quoted 3 lines

There is use of rte_smp_*mb in those functions as well. Those need to be fixed

first and then look at these.

Looks like you want __atomic_thread_fence() here.

In the rte_eal_remote_launch case, all the memory operations before the API call need to be visible to the worker. If this is the only requirement, we can use the function pointer as the guard variable and use store-release. In the eal_thread_loop function we could do load-acquire on the function pointer.

I do not think that there is a requirement to ensure that the memory operations after the API call do not happen before the worker thread starts running the function (As there is no guarantee on when the worker thread will run. If the main thread needs to know if the worker thread is running explicit hand-shaking needs to happen).

Quoted 5 lines

The rte_eal_wait_lcore API needs to ensure that the memory operations in the worker are visible to the main. rte_eal_wait_lcore and eal_thread_loop are synchronizing using lcore_config[worker_id].state. I need to understand what else 'state' is used for. If there are no issues, we can do a store-release on 'state' in eal_thread_loop and a load-

acquire in rte_eal_wait_lcore.

Quoted 3 lines

So, we do not have to use the __atomic_thread_fence.

#13 lore raw

PATCH 2/2 app/test: collect perf data after worker threads exit

In reply to Feifei Wang

The measure_perf function should be excuted after worker threads exit to collect correct perf data. Otherwise, while workers are running, the main thread may get incomplete data from workers.

In the meanwhile, remove unnecessary barrier in the test. For signal variables "ldata.done" and "ldata.start", no operations should keep the order that being executed after them. So the wmb after them can be moved.

Fixes 16a277a24c9f ("test/trace: add performance test cases")
---
 app/test/test_trace_perf.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
app/test/test_trace_perf.c +2 -3
index e1ad8e6f5..46ae7d807 100644
@@ -79,7 +79,6 @@ signal_workers_to_finish(struct test_data *data)
 
 	for (workers = 0; workers < data->nb_workers; workers++) {
 		data->ldata[workers].done = 1;
-		rte_smp_wmb();
 	}
 }
 
@@ -102,7 +101,6 @@ worker_fn_##func(void *arg) \
 { \
 	struct lcore_data *ldata = arg; \
 	ldata->started = 1; \
-	rte_smp_wmb(); \
 	__worker_##func(ldata); \
 	return 0; \
 }
@@ -137,11 +135,12 @@ run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
 
 	wait_till_workers_are_ready(data);
 	rte_delay_ms(100); /* Wait for some time to accumulate the stats */
-	measure_perf(str, data);
 	signal_workers_to_finish(data);
 
 	RTE_LCORE_FOREACH_WORKER(id)
 		rte_eal_wait_lcore(id);
+
+	measure_perf(str, data);
 }
 
 static int
Signature · 1 line
2.17.1
#14 lore raw

v3 cover remove smp barriers in app/test

In reply to Feifei Wang

For smp barriers in app/test, remove the unnecessary barriers and fix some bugs.

v2:
1. remove ring stress test patch for that Honnappa will do some changes for rte_remote_launch APIs

v3:
1. change the patch tag (Jerin)

Feifei Wang (1):
test/trace: collect perf data after worker threads exit

 app/test/test_trace_perf.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
Signature · 1 line
2.25.1
#15 lore raw

PATCH v3 1/1 test/trace: collect perf data after worker threads exit

The measure_perf function should be executed after worker threads exit to collect correct perf data. Otherwise, while workers are running, the main thread may get incomplete data from workers.

In the meanwhile, remove unnecessary barrier in the test. For signal variables "ldata.done" and "ldata.start", no operations should keep the order that being executed after them. So the wmb after them can be moved.

Fixes 16a277a24c9f ("test/trace: add performance test cases")
---
 app/test/test_trace_perf.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
app/test/test_trace_perf.c +2 -3
index e1ad8e6f5..46ae7d807 100644
@@ -79,7 +79,6 @@ signal_workers_to_finish(struct test_data *data)
 
 	for (workers = 0; workers < data->nb_workers; workers++) {
 		data->ldata[workers].done = 1;
-		rte_smp_wmb();
 	}
 }
 
@@ -102,7 +101,6 @@ worker_fn_##func(void *arg) \
 { \
 	struct lcore_data *ldata = arg; \
 	ldata->started = 1; \
-	rte_smp_wmb(); \
 	__worker_##func(ldata); \
 	return 0; \
 }
@@ -137,11 +135,12 @@ run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
 
 	wait_till_workers_are_ready(data);
 	rte_delay_ms(100); /* Wait for some time to accumulate the stats */
-	measure_perf(str, data);
 	signal_workers_to_finish(data);
 
 	RTE_LCORE_FOREACH_WORKER(id)
 		rte_eal_wait_lcore(id);
+
+	measure_perf(str, data);
 }
 
 static int
Signature · 1 line
2.25.1
#16 lore raw

v3 cover remove smp barriers in app/test

In reply to Feifei Wang

For smp barriers in app/test, remove the unnecessary barriers and fix some bugs.

v2:
1. remove ring stress test patch for that Honnappa will do some changes for rte_remote_launch APIs

v3:
1. change the patch tag (Jerin)

Feifei Wang (1):
test/trace: collect perf data after worker threads exit

 app/test/test_trace_perf.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
Signature · 1 line
2.25.1
#17 lore raw

PATCH v3 1/1 test/trace: collect perf data after worker threads exit

The measure_perf function should be executed after worker threads exit to collect correct perf data. Otherwise, while workers are running, the main thread may get incomplete data from workers.

In the meanwhile, remove unnecessary barrier in the test. For signal variables "ldata.done" and "ldata.start", no operations should keep the order that being executed after them. So the wmb after them can be moved.

Fixes 16a277a24c9f ("test/trace: add performance test cases")
---
 app/test/test_trace_perf.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
app/test/test_trace_perf.c +2 -3
index e1ad8e6f5..46ae7d807 100644
@@ -79,7 +79,6 @@ signal_workers_to_finish(struct test_data *data)
 
 	for (workers = 0; workers < data->nb_workers; workers++) {
 		data->ldata[workers].done = 1;
-		rte_smp_wmb();
 	}
 }
 
@@ -102,7 +101,6 @@ worker_fn_##func(void *arg) \
 { \
 	struct lcore_data *ldata = arg; \
 	ldata->started = 1; \
-	rte_smp_wmb(); \
 	__worker_##func(ldata); \
 	return 0; \
 }
@@ -137,11 +135,12 @@ run_test(const char *str, lcore_function_t f, struct test_data *data, size_t sz)
 
 	wait_till_workers_are_ready(data);
 	rte_delay_ms(100); /* Wait for some time to accumulate the stats */
-	measure_perf(str, data);
 	signal_workers_to_finish(data);
 
 	RTE_LCORE_FOREACH_WORKER(id)
 		rte_eal_wait_lcore(id);
+
+	measure_perf(str, data);
 }
 
 static int
Signature · 1 line
2.25.1
#18 lore raw

Re: [dpdk-dev] [PATCH v3 1/1] test/trace: collect perf data after worker threads exit

Quoted 13 lines · Feifei Wang

The measure_perf function should be executed after worker threads exit to collect correct perf data. Otherwise, while workers are running, the main thread may get incomplete data from workers.

In the meanwhile, remove unnecessary barrier in the test. For signal variables "ldata.done" and "ldata.start", no operations should keep the order that being executed after them. So the wmb after them can be moved.

Fixes 16a277a24c9f ("test/trace: add performance test cases")
Quoted 6 lines

Applied, thanks.

Signature · 1 line
David Marchand