Skip to contentSkip to conversation

app/dma-perf: introduce dma-perf application

PATCH New Cheng Jiang 8 participants 52 messages Apr 20, 2023 – Jul 7, 2023 Arm/Amlogic Meson SoC support Page 2 of 2

lore
Share & tools
lore
#23 lore raw
In reply to Anoob Joseph

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 379 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 2:32 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

Please see inline.

Thanks,
Anoob

Quoted 4 lines

-----Original Message-----
From: Jiang, Cheng1 <[email protected]>
Sent: Friday, June 16, 2023 8:26 AM
To: Anoob Joseph <[email protected]>

Quoted 8 lines

Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf

application

Quoted 15 lines

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 7 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Thursday, June 15, 2023 11:48 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 14 lines
Quoted 5 lines

<[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf

application

Quoted 8 lines

Hi Cheng,

Please see inline.

Thanks,
Anoob

Quoted 10 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Jiang, Cheng1 <[email protected]>
Sent: Thursday, June 15, 2023 7:36 PM
To: Anoob Joseph <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 2 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla
<[email protected]>; Ma, WenwuX

<[email protected]>;

Quoted 19 lines
Quoted 19 lines
Quoted 3 lines

Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf

application

Quoted 15 lines

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 7 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Thursday, June 15, 2023 4:45 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Quoted 22 lines
Quoted 3 lines

Chenbo

Quoted 2 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX

<[email protected]>;

Quoted 18 lines
Quoted 18 lines
Quoted 3 lines

Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf

application

Quoted 14 lines

Hi Cheng,

Please see inline.

Thanks,
Anoob

Quoted 6 lines

-----Original Message-----
From: Jiang, Cheng1 <[email protected]>
Sent: Thursday, June 15, 2023 1:31 PM
To: Anoob Joseph <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo

<[email protected]>;

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Amit

Quoted 3 lines
Quoted 3 lines

Prakash Shukla <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 2 lines
Quoted 2 lines

YuanX

Quoted 1 line

<[email protected]>; He, Xingguang

<[email protected]>

Quoted 18 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi,

Thanks for your comments, the replies are inline.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Thanks,
Cheng

Quoted 4 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Thursday, June 15, 2023 1:22 PM
To: Jiang, Cheng1 <[email protected]>;

[email protected];

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Richardson, Bruce <[email protected]>;
[email protected]; Xia, Chenbo

<[email protected]>;

Quoted 14 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Amit

Quoted 3 lines
Quoted 3 lines

Prakash Shukla <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 2 lines
Quoted 2 lines

YuanX

Quoted 1 line

<[email protected]>; He, Xingguang

<[email protected]>

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi,

Thanks for working on the comments. Few more top level

comment

Quoted 23 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

inline.

Quoted 9 lines
Quoted 9 lines

Thanks,
Anoob

Quoted 5 lines

-----Original Message-----
From: Cheng Jiang <[email protected]>
Sent: Tuesday, June 13, 2023 10:02 AM
To: [email protected]; [email protected]; [email protected]; [email protected]; Amit

Prakash

Quoted 12 lines
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines

Shukla

Quoted 1 line
Quoted 1 line

<[email protected]>; Anoob Joseph

<[email protected]>

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 6 lines
Quoted 6 lines

Cheng Jiang <[email protected]>
Subject: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

External Email

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

------------------------------------------------------------------
--
-- There are many high-performance DMA devices supported in

DPDK

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before

integrating

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

DMA into applications, developers need to know the

performance

Quoted 8 lines
Quoted 8 lines

of

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

these DMA devices in various scenarios and the performance of

CPUs

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios

automatically

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 1 line

with a pre- set config file. Memory Copy performance test are

supported for now.

Quoted 25 lines

---
v6:
improved code based on Anoob's comments; fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build               |   1 +
 app/test-dma-perf/benchmark.c | 477

++++++++++++++++++++++++++++

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/config.ini | 59 ++++

 app/test-dma-perf/main.c      | 569
++++++++++++++++++++++++++++++++++
 app/test-dma-perf/main.h      |  69 +++++
 app/test-dma-perf/meson.build |  17 +
 6 files changed, 1192 insertions(+)  create mode 100644

app/test-dma-perf/benchmark.c create mode 100644 app/test-dma-perf/config.ini create mode 100644 app/test-

dma-

Quoted 68 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

perf/main.c create mode 100644 app/test-dma-perf/main.h

create

Quoted 34 lines
Quoted 34 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines

mode
100644 app/test-dma-perf/meson.build

app/meson.build index renamed from app/meson.build +0 -0

74d2420f67..4fc1a83eba 100644

app/meson.build +1 -0
@@ -19,6 +19,7 @@ apps = [
         'test-cmdline',
         'test-compress-perf',
         'test-crypto-perf',
+        'test-dma-perf',
         'test-eventdev',
         'test-fib',
         'test-flow-perf',
app/test-dma- renamed from app/test-dma-perf/benchmark.c +0 -0
Quoted 1 line

perf/benchmark.c new file mode 100644 index

0000000000..bc1ca82297

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
app/test-dma-perf/benchmark.c new file +13 -0
@@ -0,0 +1,477 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Intel Corporation  */
+
+#include <inttypes.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <rte_time.h>
+#include <rte_mbuf.h>
+#include <rte_dmadev.h>
+#include <rte_malloc.h>
+#include <rte_lcore.h>
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c line 21

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+
+#include "main.h"
+
+#define MAX_DMA_CPL_NB 255
+
+#define TEST_WAIT_U_SECOND 10000
+
+#define CSV_LINE_DMA_FMT "Scenario

%u,%u,%s,%u,%u,%.2lf,%"

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

PRIu64

Quoted 2 lines
Quoted 2 lines

",%.3lf,%.3lf\n"
+#define CSV_LINE_CPU_FMT "Scenario

%u,%u,NA,%u,%u,%.2lf,%"

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

PRIu64

Quoted 18 lines
Quoted 18 lines

",%.3lf,%.3lf\n"

app/test-dma-perf/benchmark.c new file +17 -0
+
+struct worker_info {
+	bool ready_flag;
+	bool start_flag;
+	bool stop_flag;
+	uint32_t total_cpl;
+	uint32_t test_cpl;
+};
+
+struct lcore_params {
+	uint8_t scenario_id;
+	unsigned int lcore_id;
+	char *dma_name;
+	uint16_t worker_id;
+	uint16_t dev_id;
+	uint32_t nr_buf;
+	uint16_t kick_batch;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+	uint32_t buf_size;
+	uint16_t test_secs;
+	struct rte_mbuf **srcs;
+	struct rte_mbuf **dsts;
+	struct worker_info worker_info;
+};
+
+static struct rte_mempool *src_pool; static struct

rte_mempool

Quoted 43 lines
Quoted 32 lines
Quoted 32 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines
app/test-dma-perf/benchmark.c new file +3 -0
+*dst_pool;
+
+static volatile struct lcore_params

*worker_params[MAX_WORKER_NB];

Quoted 2 lines

app/test-dma-perf/benchmark.c

Quoted 2 lines
app/test-dma-perf/benchmark.c new file +2 -0
+
+#define PRINT_ERR(...) print_err(__func__, __LINE__,

__VA_ARGS__)

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
app/test-dma-perf/benchmark.c new file +12 -0
+
+static inline int
+__rte_format_printf(3, 4)
+print_err(const char *func, int lineno, const char *format, ...) {
+	va_list ap;
+	int ret;
+
+	ret = fprintf(stderr, "In %s:%d - ", func, lineno);
+	va_start(ap, format);
+	ret += vfprintf(stderr, format, ap);
+	va_end(ap);
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +6 -0
+	return ret;
+}
+
+static inline void
+calc_result(uint32_t buf_size, uint32_t nr_buf, uint16_t
+nb_workers,

uint16_t test_secs,

app/test-dma-perf/benchmark.c new file +1 -0
+				uint32_t total_cnt, float *memory,

uint32_t

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines

*ave_cycle,

app/test-dma-perf/benchmark.c new file +2 -0
+				float *bandwidth, float *mops) {
+	*memory = (float)(buf_size * (nr_buf / nb_workers) * 2) /

(1024

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

+*
1024);

app/test-dma-perf/benchmark.c new file +2 -0
+	*ave_cycle = test_secs * rte_get_timer_hz() / total_cnt;
+	*bandwidth = (buf_size * 8 * (rte_get_timer_hz() /

(float)*ave_cycle)) / 1000000000;

[Anoob] The above calculation may not yield actual results. 'ave_cycle'

would

Quoted 3 lines

get converted to integer and then bandwidth would be allowed to report only very few values. Instead, we can do the calculation directly like,

Quoted 12 lines
Quoted 5 lines

*bandwidth = ((float)buf_size * 8 * total_cnt / test_secs) /

1000000000;
	*mops = (float)total_cnt / test_secs / 1000000;

Same issue is there with below calculation as well. Please check.

[Cheng] Yes, I've noticed as well. Dengdui also mentioned this in his comments. I will address this issue in v7. Thank you very much.

Quoted 3 lines

Side note: in bandwidth calculation, shouldn't we be dividing by 1024*1024*1024? I've just carried the calculation that you used. Feel free

to

Quoted 10 lines
Quoted 1 line

correct as required.

[Cheng] The unit I'm using in my calculations is Gb/s (Gigabits per second), which is based on the decimal system. Therefore, I use the factor of 1000^3 (or 1,000,000,000).
The method you mentioned, dividing by 1024^3, is typically used when calculating GiB/s (Gibibits per second), a binary-based unit.
I think both methods are acceptable as long as the units and calculation methods correspond.
What do you think?

Quoted 1 line

[Anoob] You are right. Existing logic is correct. Thanks for the explanation.

[Cheng] sure, no problem. Feel free to contact me if you have any more questions.

Quoted 1,495 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

app/test-dma-perf/benchmark.c

Quoted 10 lines
app/test-dma-perf/benchmark.c new file +5 -0
+	*mops = (float)rte_get_timer_hz() / *ave_cycle / 1000000; }
+
+static void
+output_result(uint8_t scenario_id, uint32_t lcore_id, char
+*dma_name,

uint64_t ave_cycle,

app/test-dma-perf/benchmark.c new file +1 -0
+			uint32_t buf_size, uint32_t nr_buf, float

memory,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

app/test-dma-perf/benchmark.c

Quoted 7 lines
app/test-dma-perf/benchmark.c new file +7 -0
+			float bandwidth, float mops, bool is_dma) {
+	if (is_dma)
+		printf("lcore %u, DMA %s:\n", lcore_id, dma_name);
+	else
+		printf("lcore %u\n", lcore_id);
+
+	printf("average cycles/op: %" PRIu64 ", buffer size: %u,

nr_buf:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

app/test-dma-perf/benchmark.c

Quoted 5 lines

+%u,
memory: %.2lfMB, frequency: %" PRIu64 ".\n",

app/test-dma-perf/benchmark.c new file +1 -0
+			ave_cycle, buf_size, nr_buf, memory,

rte_get_timer_hz());

app/test-dma-perf/benchmark.c new file +1 -0
+	printf("Average bandwidth: %.3lfGbps, MOps: %.3lf\n",

bandwidth,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/benchmark.c

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+mops);
+
+	if (is_dma)
+		snprintf(output_str[lcore_id],

MAX_OUTPUT_STR_LEN,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines

CSV_LINE_DMA_FMT,

app/test-dma-perf/benchmark.c new file +2 -0
+			scenario_id, lcore_id, dma_name, buf_size,
+			nr_buf, memory, ave_cycle, bandwidth,

mops);

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

app/test-dma-perf/benchmark.c

Quoted 2 lines
app/test-dma-perf/benchmark.c new file +2 -0
+	else
+		snprintf(output_str[lcore_id],

MAX_OUTPUT_STR_LEN,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines

CSV_LINE_CPU_FMT,

app/test-dma-perf/benchmark.c new file +2 -0
+			scenario_id, lcore_id, buf_size,
+			nr_buf, memory, ave_cycle, bandwidth,

mops); }

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

app/test-dma-perf/benchmark.c

Quoted 5 lines
app/test-dma-perf/benchmark.c new file +5 -0
+
+static inline void
+cache_flush_buf(__maybe_unused struct rte_mbuf **array,
+		__maybe_unused uint32_t buf_size,
+		__maybe_unused uint32_t nr_buf) { #ifdef

RTE_ARCH_X86_64

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
app/test-dma-perf/benchmark.c new file +7 -0
+	char *data;
+	struct rte_mbuf **srcs = array;
+	uint32_t i, offset;
+
+	for (i = 0; i < nr_buf; i++) {
+		data = rte_pktmbuf_mtod(srcs[i], char *);
+		for (offset = 0; offset < buf_size; offset += 64)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+			__builtin_ia32_clflush(data + offset);
+	}
+#endif
+}
+
+/* Configuration of device. */
+static void
+configure_dmadev_queue(uint32_t dev_id, uint32_t ring_size)

{

Quoted 191 lines
Quoted 69 lines
Quoted 69 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

app/test-dma-perf/benchmark.c

Quoted 10 lines
app/test-dma-perf/benchmark.c new file +10 -0
+	uint16_t vchan = 0;
+	struct rte_dma_info info;
+	struct rte_dma_conf dev_config = { .nb_vchans = 1 };
+	struct rte_dma_vchan_conf qconf = {
+		.direction = RTE_DMA_DIR_MEM_TO_MEM,
+		.nb_desc = ring_size
+	};
+
+	if (rte_dma_configure(dev_id, &dev_config) != 0)
+		rte_exit(EXIT_FAILURE, "Error with dma

configure.\n");

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines
app/test-dma-perf/benchmark.c new file +3 -0
+
+	if (rte_dma_vchan_setup(dev_id, vchan, &qconf) != 0)
+		rte_exit(EXIT_FAILURE, "Error with queue

configuration.\n");

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
app/test-dma-perf/benchmark.c new file +4 -0
+
+	rte_dma_info_get(dev_id, &info);
+	if (info.nb_vchans != 1)
+		rte_exit(EXIT_FAILURE, "Error, no configured queues

reported on device id. %u\n",

app/test-dma-perf/benchmark.c new file +10 -0
+				dev_id);
+
+	if (rte_dma_start(dev_id) != 0)
+		rte_exit(EXIT_FAILURE, "Error with dma start.\n"); }
+
+static int
+config_dmadevs(struct test_configure *cfg) {
+	uint32_t ring_size = cfg->ring_size.cur;
+	struct lcore_dma_map_t *ldm = &cfg->lcore_dma_map;
+	uint32_t nb_workers = ldm->cnt;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+	uint32_t i;
+	int dev_id;
+	uint16_t nb_dmadevs = 0;
+	char *dma_name;
+
+	for (i = 0; i < ldm->cnt; i++) {
+		dma_name = ldm->dma_names[i];
+		dev_id =

rte_dma_get_dev_id_by_name(dma_name);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
app/test-dma-perf/benchmark.c new file +2 -0
+		if (dev_id == -1) {
+			fprintf(stderr, "Error: Fail to find DMA %s.\n",

dma_name);

app/test-dma-perf/benchmark.c new file +3 -0
+			goto end;
+		}
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+		ldm->dma_ids[i] = dev_id;
+		configure_dmadev_queue(dev_id, ring_size);
+		++nb_dmadevs;
+	}
+
+end:
+	if (nb_dmadevs < nb_workers) {
+		printf("Not enough dmadevs (%u) for all workers

(%u).\n",

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

app/test-dma-perf/benchmark.c

Quoted 12 lines

nb_dmadevs, nb_workers);

app/test-dma-perf/benchmark.c new file +11 -0
+		return -1;
+	}
+
+	printf("Number of used dmadevs: %u.\n", nb_dmadevs);
+
+	return 0;
+}
+
+#define POLL_MAX 1000
+
+

[Anoob] Extra blank line. You can consider removing.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

[Cheng] sure, sorry for the miss.

Quoted 3 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines
app/test-dma-perf/benchmark.c new file +2 -0
+static inline void
+do_dma_submit_and_poll(uint16_t dev_id, uint64_t

*async_cnt,

Quoted 116 lines
Quoted 116 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+			volatile struct worker_info *worker_info) {
+	int ret;
+	uint16_t nr_cpl;
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+	ret = rte_dma_submit(dev_id, 0);
+	if (ret < 0) {
+		rte_dma_stop(dev_id);
+		rte_dma_close(dev_id);
+		rte_exit(EXIT_FAILURE, "Error with dma submit.\n");
+	}
+
+	nr_cpl = rte_dma_completed(dev_id, 0,

MAX_DMA_CPL_NB, NULL,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

app/test-dma-perf/benchmark.c

Quoted 9 lines

NULL);

app/test-dma-perf/benchmark.c new file +8 -0
+	*async_cnt -= nr_cpl;
+	worker_info->total_cpl += nr_cpl; }
+
+static inline int
+do_dma_mem_copy(void *p)
+{
+	const uint16_t *para_idx = (uint16_t *)p;
+	volatile struct lcore_params *para =

worker_params[*para_idx];

Quoted 18 lines
Quoted 1 line
Quoted 1 line

app/test-dma-perf/benchmark.c

Quoted 1 line
app/test-dma-perf/benchmark.c new file +1 -0
+	volatile struct worker_info *worker_info = &(para-

worker_info);

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
app/test-dma-perf/benchmark.c new file +16 -0
+	const uint16_t dev_id = para->dev_id;
+	const uint32_t nr_buf = para->nr_buf;
+	const uint16_t kick_batch = para->kick_batch;
+	const uint32_t buf_size = para->buf_size;
+	struct rte_mbuf **srcs = para->srcs;
+	struct rte_mbuf **dsts = para->dsts;
+	uint16_t nr_cpl;
+	uint64_t async_cnt = 0;
+	uint32_t i;
+	uint32_t poll_cnt = 0;
+	int ret;
+
+	worker_info->stop_flag = false;
+	worker_info->ready_flag = true;
+
+	while (!worker_info->start_flag)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +6 -0
+		;
+
+	while (1) {
+		for (i = 0; i < nr_buf; i++) {
+dma_copy:
+			ret = rte_dma_copy(dev_id, 0,

rte_pktmbuf_iova(srcs[i]),

app/test-dma-perf/benchmark.c new file +1 -0
+				rte_pktmbuf_iova(dsts[i]), buf_size,

0);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines
app/test-dma-perf/benchmark.c new file +3 -0
+			if (unlikely(ret < 0)) {
+				if (ret == -ENOSPC) {
+

do_dma_submit_and_poll(dev_id,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

app/test-dma-perf/benchmark.c

Quoted 6 lines

&async_cnt, worker_info);

app/test-dma-perf/benchmark.c new file +5 -0
+					goto dma_copy;
+				} else {
+					/* Error exit */
+					rte_dma_stop(dev_id);
+					rte_exit(EXIT_FAILURE,

"DMA

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

enqueue failed\n");

app/test-dma-perf/benchmark.c new file +6 -0
+				}
+			}
+			async_cnt++;
+
+			if ((async_cnt % kick_batch) == 0)
+				do_dma_submit_and_poll(dev_id,

&async_cnt, worker_info);

app/test-dma-perf/benchmark.c new file +9 -0
+		}
+
+		if (worker_info->stop_flag)
+			break;
+	}
+
+	rte_dma_submit(dev_id, 0);
+	while ((async_cnt > 0) && (poll_cnt++ < POLL_MAX)) {
+		nr_cpl = rte_dma_completed(dev_id, 0,

MAX_DMA_CPL_NB, NULL, NULL);

app/test-dma-perf/benchmark.c new file +3 -0
+		async_cnt -= nr_cpl;
+	}
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+	return 0;
+}
+
+static inline int
+do_cpu_mem_copy(void *p)
+{
+	const uint16_t *para_idx = (uint16_t *)p;
+	volatile struct lcore_params *para =

worker_params[*para_idx];

Quoted 25 lines
Quoted 1 line
Quoted 1 line

app/test-dma-perf/benchmark.c

Quoted 1 line
app/test-dma-perf/benchmark.c new file +1 -0
+	volatile struct worker_info *worker_info = &(para-

worker_info);

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
app/test-dma-perf/benchmark.c new file +16 -0
+	const uint32_t nr_buf = para->nr_buf;
+	const uint32_t buf_size = para->buf_size;
+	struct rte_mbuf **srcs = para->srcs;
+	struct rte_mbuf **dsts = para->dsts;
+	uint32_t i;
+
+	worker_info->stop_flag = false;
+	worker_info->ready_flag = true;
+
+	while (!worker_info->start_flag)
+		;
+
+	while (1) {
+		for (i = 0; i < nr_buf; i++) {
+			/* copy buffer form src to dst */
+			rte_memcpy((void

*)(uintptr_t)rte_mbuf_data_iova(dsts[i]),

app/test-dma-perf/benchmark.c new file +1 -0
+				(void

*)(uintptr_t)rte_mbuf_data_iova(srcs[i]),

app/test-dma-perf/benchmark.c new file +4 -0
+				(size_t)buf_size);
+			worker_info->total_cpl++;
+		}
+		if (worker_info->stop_flag)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+			break;
+	}
+
+	return 0;
+}
+
+static int
+setup_memory_env(struct test_configure *cfg, struct

rte_mbuf

Quoted 58 lines
Quoted 58 lines
Quoted 58 lines
Quoted 11 lines
Quoted 11 lines

***srcs,

Quoted 10 lines

app/test-dma-perf/benchmark.c

Quoted 10 lines
app/test-dma-perf/benchmark.c new file +10 -0
+			struct rte_mbuf ***dsts)
+{
+	unsigned int buf_size = cfg->buf_size.cur;
+	unsigned int nr_sockets;
+	uint32_t nr_buf = cfg->nr_buf;
+
+	nr_sockets = rte_socket_count();
+	if (cfg->src_numa_node >= nr_sockets ||
+		cfg->dst_numa_node >= nr_sockets) {
+		printf("Error: Source or destination numa exceeds

the acture

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

app/test-dma-perf/benchmark.c

Quoted 5 lines

numa nodes.\n");

app/test-dma-perf/benchmark.c new file +4 -0
+		return -1;
+	}
+
+	src_pool =

rte_pktmbuf_pool_create("Benchmark_DMA_SRC",

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

app/test-dma-perf/benchmark.c

Quoted 7 lines
app/test-dma-perf/benchmark.c new file +7 -0
+			nr_buf, /* n == num elements */
+			64,  /* cache size */
+			0,   /* priv size */
+			buf_size + RTE_PKTMBUF_HEADROOM,
+			cfg->src_numa_node);
+	if (src_pool == NULL) {
+		PRINT_ERR("Error with source mempool

creation.\n");

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/benchmark.c

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+		return -1;
+	}
+
+	dst_pool =

rte_pktmbuf_pool_create("Benchmark_DMA_DST",

Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

app/test-dma-perf/benchmark.c

Quoted 2 lines
app/test-dma-perf/benchmark.c new file +2 -0
+			nr_buf, /* n == num elements */
+			64,  /* cache size */

[Anoob] We do not alloc or free pointers in the datapath, right? So why bother with cache?

[Cheng] Yes, you are right, the cache size is not necessary here, I'll fix it in the next version.

Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

app/test-dma-perf/benchmark.c

Quoted 5 lines
app/test-dma-perf/benchmark.c new file +5 -0
+			0,   /* priv size */
+			buf_size + RTE_PKTMBUF_HEADROOM,
+			cfg->dst_numa_node);
+	if (dst_pool == NULL) {
+		PRINT_ERR("Error with destination mempool

creation.\n");

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/benchmark.c

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+		return -1;
+	}
+
+	*srcs = rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *),

0);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

app/test-dma-perf/benchmark.c

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+	if (*srcs == NULL) {
+		printf("Error: srcs malloc failed.\n");
+		return -1;
+	}

[Anoob] Are we freeing these memory? The ones allocated with

Quoted 10 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

rte_malloc.

[Cheng] yes, we freed the memory in the end of

mem_copy_benchmark()

Quoted 3 lines
Quoted 1 line

when we finished the test.

[Anoob] I think we are not freeing this mem. In the place where we

free

Quoted 2 lines

all

Quoted 1 line
Quoted 1 line

mem, we do free all objects to mempool as well as the mempools.

But

Quoted 4 lines
Quoted 4 lines

this

Quoted 3 lines
Quoted 3 lines

memory is to hold the pointers, right? Is that getting freed anywhere?

Also, in the mem clearing paths, do we need to clear the static

variables

Quoted 7 lines
Quoted 7 lines

(ie,

Quoted 1 line
Quoted 1 line

set srcs, src_pool, dsts, dst_pool to NULL) so that there won't be any

scope

Quoted 4 lines
Quoted 2 lines

for any double free.

[Cheng] My apologies for the misunderstanding earlier. I now

understand

Quoted 10 lines
Quoted 1 line
Quoted 1 line

your point that you are right, the memory used to store the pointers is

not

Quoted 4 lines
Quoted 2 lines

being freed. I will fix this issue in the next version. Regarding the static variables you mentioned, I agree with your view that they should be

cleared.

Quoted 1 line

I will address this in the upcoming version as well. Thank you very much

for

Quoted 3 lines
Quoted 3 lines

the feedback. It is greatly appreciated.

In addition, I think we also need to nullify these variables when

initializing

Quoted 31 lines
Quoted 7 lines
Quoted 1 line

them to ensure safety and standardization of use. What do you think?

[Anoob] Since these are static variables, it is probably okay to skip the init part. But when we use it, we should clear it after use.

Please check above. I've posted one more comment. In case you missed.

[Cheng] sure, thanks for your advice, I'll clear it after use in the next version, thanks.

Quoted 20 lines
Quoted 20 lines

Thanks!

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/benchmark.c

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +2 -0
+
+	*dsts = rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *),

0);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

app/test-dma-perf/benchmark.c

Quoted 6 lines
app/test-dma-perf/benchmark.c new file +6 -0
+	if (*dsts == NULL) {
+		printf("Error: dsts malloc failed.\n");
+		return -1;
+	}
+
+	if (rte_mempool_get_bulk(src_pool, (void **)*srcs, nr_buf)

!= 0) {

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/benchmark.c

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+		printf("get src mbufs failed.\n");
+		return -1;
+	}
+	if (rte_mempool_get_bulk(dst_pool, (void **)*dsts, nr_buf)

!= 0) {

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

app/test-dma-perf/benchmark.c

Quoted 9 lines
app/test-dma-perf/benchmark.c new file +9 -0
+		printf("get dst mbufs failed.\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+void
+mem_copy_benchmark(struct test_configure *cfg, bool

is_dma)

Quoted 158 lines

{

Quoted 143 lines
Quoted 143 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
app/test-dma-perf/benchmark.c new file +9 -0
+	uint16_t i;
+	uint32_t offset;
+	unsigned int lcore_id = 0;
+	struct rte_mbuf **srcs = NULL, **dsts = NULL;
+	struct lcore_dma_map_t *ldm = &cfg->lcore_dma_map;
+	unsigned int buf_size = cfg->buf_size.cur;
+	uint16_t kick_batch = cfg->kick_batch.cur;
+	uint32_t nr_buf = cfg->nr_buf = (cfg->mem_size.cur * 1024 *
+1024) /

(cfg->buf_size.cur * 2);

app/test-dma-perf/benchmark.c new file +22 -0
+	uint16_t nb_workers = ldm->cnt;
+	uint16_t test_secs = cfg->test_secs;
+	float memory;
+	uint32_t avg_cycles = 0;
+	float mops;
+	float bandwidth;
+
+	if (setup_memory_env(cfg, &srcs, &dsts) < 0)
+		goto out;
+
+	if (is_dma)
+		if (config_dmadevs(cfg) < 0)
+			goto out;
+
+	if (cfg->cache_flush) {
+		cache_flush_buf(srcs, buf_size, nr_buf);
+		cache_flush_buf(dsts, buf_size, nr_buf);
+		rte_mb();
+	}
+
+	printf("Start testing....\n");
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +5 -0
+	for (i = 0; i < nb_workers; i++) {
+		lcore_id = ldm->lcores[i];
+		offset = nr_buf / nb_workers * i;
+
+		worker_params[i] = rte_malloc(NULL, sizeof(struct

lcore_params), 0);

app/test-dma-perf/benchmark.c new file +2 -0
+		if (!worker_params[i]) {
+			printf("lcore parameters malloc failure for

lcore

Quoted 15 lines
Quoted 6 lines
Quoted 5 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines

%d\n", lcore_id);

app/test-dma-perf/benchmark.c new file +2 -0
+			break;
+		}

[Anoob] Are we freeing the above memory?

Quoted 8 lines

[Cheng] sorry, I missed that, I'll add worker_params memory free in the next version, thanks.

Quoted 5 lines

app/test-dma-perf/benchmark.c

Quoted 5 lines
app/test-dma-perf/benchmark.c new file +2 -0
+		if (is_dma) {
+			worker_params[i]->dma_name = ldm-
Quoted 1 line

dma_names[i];

app/test-dma-perf/benchmark.c new file +1 -0
+			worker_params[i]->dev_id = ldm-

dma_ids[i];

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
app/test-dma-perf/benchmark.c new file +4 -0
+			worker_params[i]->kick_batch = kick_batch;
+		}
+		worker_params[i]->worker_id = i;
+		worker_params[i]->nr_buf = (uint32_t)(nr_buf /

nb_workers);

app/test-dma-perf/benchmark.c new file +1 -0
+		worker_params[i]->buf_size = buf_size;
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+		worker_params[i]->test_secs = test_secs;
+		worker_params[i]->srcs = srcs + offset;
+		worker_params[i]->dsts = dsts + offset;
+		worker_params[i]->scenario_id = cfg->scenario_id;
+		worker_params[i]->lcore_id = lcore_id;
+
+		if (is_dma)
+

rte_eal_remote_launch(do_dma_mem_copy, (void

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/benchmark.c

Quoted 3 lines

*)(&i), lcore_id);

app/test-dma-perf/benchmark.c new file +2 -0
+		else
+			rte_eal_remote_launch(do_cpu_mem_copy,

(void

Quoted 31 lines
Quoted 7 lines
Quoted 7 lines

app/test-dma-perf/benchmark.c

Quoted 7 lines

*)(&i), lcore_id);

app/test-dma-perf/benchmark.c new file +6 -0
+	}
+
+	while (1) {
+		bool ready = true;
+		for (i = 0; i < nb_workers; i++) {
+			if (worker_params[i]-

worker_info.ready_flag ==

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines

false) {

app/test-dma-perf/benchmark.c new file +19 -0
+				ready = 0;
+				break;
+			}
+		}
+		if (ready)
+			break;
+	}
+
+	for (i = 0; i < nb_workers; i++)
+		worker_params[i]->worker_info.start_flag = true;
+
+	usleep(TEST_WAIT_U_SECOND);
+	for (i = 0; i < nb_workers; i++)
+		worker_params[i]->worker_info.test_cpl =
+worker_params[i]->worker_info.total_cpl;
+
+	usleep(test_secs * 1000 * 1000);
+	for (i = 0; i < nb_workers; i++)
+		worker_params[i]->worker_info.test_cpl =

worker_params[i]->worker_info.total_cpl -

app/test-dma-perf/benchmark.c new file +1 -0
+						worker_params[i]-
Quoted 1 line

worker_info.test_cpl;

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

app/test-dma-perf/benchmark.c

Quoted 10 lines
app/test-dma-perf/benchmark.c new file +10 -0
+
+	for (i = 0; i < nb_workers; i++)
+		worker_params[i]->worker_info.stop_flag = true;
+
+	rte_eal_mp_wait_lcore();
+
+	for (i = 0; i < nb_workers; i++) {
+		calc_result(buf_size, nr_buf, nb_workers, test_secs,
+			worker_params[i]->worker_info.test_cpl,
+			&memory, &avg_cycles, &bandwidth,

&mops);

Quoted 6 lines
Quoted 1 line
Quoted 1 line

app/test-dma-perf/benchmark.c

Quoted 1 line
app/test-dma-perf/benchmark.c new file +1 -0
+		output_result(cfg->scenario_id, worker_params[i]-

lcore_id,

Quoted 1 line
Quoted 1 line

app/test-dma-perf/benchmark.c

Quoted 1 line
app/test-dma-perf/benchmark.c new file +1 -0
+					worker_params[i]-

dma_name,

Quoted 2 lines
Quoted 2 lines

app/test-dma-perf/benchmark.c

Quoted 2 lines

avg_cycles, buf_size,

app/test-dma-perf/benchmark.c new file +1 -0
+					nr_buf / nb_workers,

memory,

Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

bandwidth, mops, is_dma);

app/test-dma-perf/benchmark.c new file +9 -0
+	}
+
+out:
+	/* free env */
+	if (srcs)
+		rte_pktmbuf_free_bulk(srcs, nr_buf);
+	if (dsts)
+		rte_pktmbuf_free_bulk(dsts, nr_buf);
+
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/benchmark.c

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+	if (src_pool)
+		rte_mempool_free(src_pool);
+	if (dst_pool)
+		rte_mempool_free(dst_pool);
+
+	if (is_dma) {
+		for (i = 0; i < nb_workers; i++) {
+			printf("Stopping dmadev %d\n", ldm-

dma_ids[i]);

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
app/test-dma-perf/benchmark.c new file +4 -0
+			rte_dma_stop(ldm->dma_ids[i]);
+		}
+	}
+}
(quoted diff) +0 -0
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/config.ini line 3

Quoted 8 lines

b/app/test-dma-perf/config.ini new file mode 100644 index 0000000000..2fd9c3c387

app/test-dma-perf/config.ini new file +3 -0
@@ -0,0 +1,59 @@
+
+; This is an example configuration file for dma-perf, which
+details the meanings of each parameter ; and instructions on

how

Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

app/test-dma-perf/config.ini line 4

Quoted 1 line
app/test-dma-perf/config.ini new file +1 -0
+to use dma-

perf.

Quoted 2 lines
Quoted 2 lines
app/test-dma-perf/config.ini new file +2 -0
+
+; Supported test types are DMA_MEM_COPY and

CPU_MEM_COPY.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
app/test-dma-perf/config.ini new file +5 -0
+
+; Parameters:
+; "mem_size" denotes the size of the memory footprint.
+; "buf_size" denotes the memory size of a single operation.
+; "dma_ring_size" denotes the dma ring buffer size. It should

be

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
app/test-dma-perf/config.ini new file +1 -0
+greater

than 64 normally.
+; "kick_batch" denotes the dma operation batch size, and

should

Quoted 69 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/config.ini

Quoted 4 lines

+be greater
than 1 normally.

app/test-dma-perf/config.ini new file +2 -0
+
+; The format for variables is

variable=first,last,increment,ADD|MUL.

Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

app/test-dma-perf/config.ini

Quoted 2 lines
app/test-dma-perf/config.ini new file +2 -0
+
+; src_numa_node is used to control the numa node where the

source

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

app/test-dma-perf/config.ini

Quoted 7 lines

memory is allocated.

app/test-dma-perf/config.ini new file +2 -0
+; dst_numa_node is used to control the numa node where the
+destination

memory is allocated.

app/test-dma-perf/config.ini new file +3 -0
+
+; cache_flush is used to determine whether or not the cache
+should be flushed, with 1 indicating to ; flush and 0 indicating to

not

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

flush.

Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/config.ini

Quoted 4 lines
app/test-dma-perf/config.ini new file +4 -0
+
+; test_seconds controls the test time of the whole case.
+
+; To use DMA for a test, please specify the "lcore_dma"

parameter.

Quoted 37 lines
Quoted 37 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/config.ini

Quoted 8 lines
app/test-dma-perf/config.ini new file +3 -0
+; If you have already set the "-l" and "-a" parameters using EAL,
+; make sure that the value of "lcore_dma" falls within their
+range of the

values.

app/test-dma-perf/config.ini new file +4 -0
+
+; To use CPU for a test, please specify the "lcore" parameter.
+; If you have already set the "-l" and "-a" parameters using EAL,
+; make sure that the value of "lcore" falls within their range of

values.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 14 lines
app/test-dma-perf/config.ini new file +3 -0
+
+; To specify a configuration file, use the "--config" flag
+followed by the path

to the file.

app/test-dma-perf/config.ini new file +3 -0
+
+; To specify a result file, use the "--result" flag followed by
+the path to the

file.

app/test-dma-perf/config.ini new file +3 -0
+; If you do not specify a result file, one will be generated with
+the same name as the configuration ; file, with the addition of
+"_result.csv" at

the end.

app/test-dma-perf/config.ini new file +2 -0
+
+[case1]

app/test-dma-perf/config.ini

Quoted 10 lines
app/test-dma-perf/config.ini new file +10 -0
+type=DMA_MEM_COPY
+mem_size=10
+buf_size=64,8192,2,MUL
+dma_ring_size=1024
+kick_batch=32
+src_numa_node=0
+dst_numa_node=0
+cache_flush=0
+test_seconds=2
+lcore_dma=lcore10@0000:00:04.2, lcore11@0000:00:04.3

[Anoob] Isn't it better if we allow user to specify DMA dev ID rather than the PCI DBDF?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

In the long run, I would expect config file to provide {core, dma_dev_id, queue_id}

Another thought is why to expose this at all? If we can restrict this perf application to have one thread only use one vchan, then application can easily create this mapping in run time. Unless you want one thread to use 2 different vchans which may not be

desirable

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

since this is a standalone perf app.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

[Cheng] Thank you for the feedback.
Here are my thoughts:
Firstly, the user may not know which device the DMA dev ID

corresponds

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to, or which NUMA node it is on. In my example, I used the CBDMA environment, so I did not specify the work queue ID. When using

DSA,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the configuration would be something like lcore10@0000:00:04.2-q0 which contains core, dma and work queue id. The reason for

exposing

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

these options is that we want the user to fully understand which

cores

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

and devices are being used so that they know exactly where the performance data is coming from. For example, performance when

cores

Quoted 1 line
Quoted 1 line
Quoted 1 line

and DMA devices are not on the same NUMA node, etc. This allows

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

testing scenario to be precise and flexible. If the application handles the mapping itself, the user loses control over the mapping and may not get the performance data they want. We believe

control

Quoted 181 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line

should be given to the user rather than the application.

[Anoob] I understand your view points. Thanks for the explanation.

[Cheng] sure, no problem.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
app/test-dma-perf/config.ini new file +5 -0
+eal_args=--in-memory --file-prefix=test
+
+[case2]
+type=CPU_MEM_COPY
+mem_size=10
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-

Quoted 8 lines
app/test-dma-perf/config.ini new file +7 -0
+buf_size=64,8192,2,MUL
+src_numa_node=0
+dst_numa_node=1
+cache_flush=0
+test_seconds=2
+lcore = 3, 4
+eal_args=--in-memory --no-pci
app/test-dma- renamed from app/test-dma-perf/main.c +0 -0

perf/main.c

Quoted 80 lines
Quoted 80 lines
Quoted 71 lines
Quoted 71 lines
Quoted 71 lines
Quoted 71 lines

new file mode 100644 index 0000000000..d65655b87b

app/test-dma-perf/main.c new file +21 -0
@@ -0,0 +1,569 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Intel Corporation  */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <getopt.h>
+#include <signal.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <sys/wait.h>
+#include <inttypes.h>
+#include <libgen.h>
+
+#include <rte_eal.h>
+#include <rte_cfgfile.h>
+#include <rte_string_fns.h>
+#include <rte_lcore.h>
+
+#include "main.h"
+
+#define CSV_HDR_FMT "Case %u : %s,lcore,DMA,buffer

size,nr_buf,memory(MB),cycle,bandwidth(Gbps),MOps\n"

app/test-dma-perf/main.c new file +45 -0
+
+#define MAX_EAL_PARAM_NB 100
+#define MAX_EAL_PARAM_LEN 1024
+
+#define DMA_MEM_COPY "DMA_MEM_COPY"
+#define CPU_MEM_COPY "CPU_MEM_COPY"
+
+#define CMDLINE_CONFIG_ARG "--config"
+#define CMDLINE_RESULT_ARG "--result"
+
+#define MAX_PARAMS_PER_ENTRY 4
+
+#define MAX_LONG_OPT_SZ 64
+
+enum {
+	TEST_TYPE_NONE = 0,
+	TEST_TYPE_DMA_MEM_COPY,
+	TEST_TYPE_CPU_MEM_COPY
+};
+
+#define MAX_TEST_CASES 16
+static struct test_configure test_cases[MAX_TEST_CASES];
+
+char output_str[MAX_WORKER_NB][MAX_OUTPUT_STR_LEN];
+
+static FILE *fd;
+
+static void
+output_csv(bool need_blankline)
+{
+	uint32_t i;
+
+	if (need_blankline) {
+		fprintf(fd, ",,,,,,,,\n");
+		fprintf(fd, ",,,,,,,,\n");
+	}
+
+	for (i = 0; i < RTE_DIM(output_str); i++) {
+		if (output_str[i][0]) {
+			fprintf(fd, "%s", output_str[i]);
+			output_str[i][0] = '\0';
+		}
+	}
+
+	fflush(fd);
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +6 -0
+}
+
+static void
+output_env_info(void)
+{
+	snprintf(output_str[0], MAX_OUTPUT_STR_LEN, "test

environment:\n");

app/test-dma-perf/main.c new file +1 -0
+	snprintf(output_str[1], MAX_OUTPUT_STR_LEN, "CPU

frequency,%"

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

app/test-dma-perf/main.c

Quoted 7 lines
app/test-dma-perf/main.c new file +7 -0
+			PRIu64 "\n", rte_get_timer_hz());
+
+	output_csv(true);
+}
+
+static void
+output_header(uint32_t case_id, struct test_configure

*case_cfg) {

Quoted 70 lines
Quoted 70 lines
Quoted 70 lines
Quoted 2 lines
Quoted 2 lines

app/test-dma-perf/main.c

Quoted 2 lines
app/test-dma-perf/main.c new file +2 -0
+	snprintf(output_str[0], MAX_OUTPUT_STR_LEN,
+			CSV_HDR_FMT, case_id, case_cfg-

test_type_str);

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
app/test-dma-perf/main.c new file +7 -0
+
+	output_csv(true);
+}
+
+static void
+run_test_case(struct test_configure *case_cfg) {
+	switch (case_cfg->test_type) {
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +8 -0
+	case TEST_TYPE_DMA_MEM_COPY:
+		mem_copy_benchmark(case_cfg, true);
+		break;
+	case TEST_TYPE_CPU_MEM_COPY:
+		mem_copy_benchmark(case_cfg, false);
+		break;
+	default:
+		printf("Unknown test type. %s\n", case_cfg-

test_type_str);

Quoted 9 lines
Quoted 9 lines

app/test-dma-perf/main.c

Quoted 9 lines
app/test-dma-perf/main.c new file +9 -0
+		break;
+	}
+}
+
+static void
+run_test(uint32_t case_id, struct test_configure *case_cfg) {
+	uint32_t i;
+	uint32_t nb_lcores = rte_lcore_count();
+	struct test_configure_entry *mem_size = &case_cfg-

mem_size;

Quoted 1 line
Quoted 1 line

app/test-dma-perf/main.c

Quoted 1 line
app/test-dma-perf/main.c new file +1 -0
+	struct test_configure_entry *buf_size = &case_cfg-

buf_size;

Quoted 1 line
Quoted 1 line

app/test-dma-perf/main.c

Quoted 1 line
app/test-dma-perf/main.c new file +1 -0
+	struct test_configure_entry *ring_size = &case_cfg-

ring_size;

Quoted 1 line
Quoted 1 line

app/test-dma-perf/main.c

Quoted 1 line
app/test-dma-perf/main.c new file +1 -0
+	struct test_configure_entry *kick_batch = &case_cfg-

kick_batch;

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
app/test-dma-perf/main.c new file +31 -0
+	struct test_configure_entry dummy = { 0 };
+	struct test_configure_entry *var_entry = &dummy;
+
+	for (i = 0; i < RTE_DIM(output_str); i++)
+		memset(output_str[i], 0, MAX_OUTPUT_STR_LEN);
+
+	if (nb_lcores <= case_cfg->lcore_dma_map.cnt) {
+		printf("Case %u: Not enough lcores.\n", case_id);
+		return;
+	}
+
+	printf("Number of used lcores: %u.\n", nb_lcores);
+
+	if (mem_size->incr != 0)
+		var_entry = mem_size;
+
+	if (buf_size->incr != 0)
+		var_entry = buf_size;
+
+	if (ring_size->incr != 0)
+		var_entry = ring_size;
+
+	if (kick_batch->incr != 0)
+		var_entry = kick_batch;
+
+	case_cfg->scenario_id = 0;
+
+	output_header(case_id, case_cfg);
+
+	for (var_entry->cur = var_entry->first; var_entry->cur <=
+var_entry-
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/main.c

Quoted 3 lines
Quoted 1 line

last;) {

app/test-dma-perf/main.c new file +2 -0
+		case_cfg->scenario_id++;
+		printf("\nRunning scenario %d\n", case_cfg-

scenario_id);

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
app/test-dma-perf/main.c new file +7 -0
+
+		run_test_case(case_cfg);
+		output_csv(false);
+
+		if (var_entry->op == OP_ADD)
+			var_entry->cur += var_entry->incr;
+		else if (var_entry->op == OP_MUL)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +8 -0
+			var_entry->cur *= var_entry->incr;
+		else
+			break;
+	}
+}
+
+static int
+parse_lcore(struct test_configure *test_case, const char

*value)

Quoted 414 lines

{

Quoted 85 lines
Quoted 85 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

app/test-dma-perf/main.c

Quoted 10 lines
app/test-dma-perf/main.c new file +4 -0
+	size_t len = strlen(value);
+	char *input = (char *) malloc((len + 1) * sizeof(char));
+	strcpy(input, value);
+	struct lcore_dma_map_t *lcore_dma_map = &(test_case-
Quoted 1 line

lcore_dma_map);

app/test-dma-perf/main.c new file +5 -0
+
+	if (test_case == NULL || value == NULL)
+		return -1;
+
+	memset(lcore_dma_map, 0, sizeof(struct

lcore_dma_map_t));

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

app/test-dma-perf/main.c

Quoted 10 lines
app/test-dma-perf/main.c new file +10 -0
+
+	char *token = strtok(input, ", ");
+	while (token != NULL) {
+		if (lcore_dma_map->cnt >= MAX_LCORE_NB) {
+			free(input);
+			return -1;
+		}
+
+		uint16_t lcore_id = atoi(token);
+		lcore_dma_map->lcores[lcore_dma_map->cnt++] =

lcore_id;

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

app/test-dma-perf/main.c

Quoted 10 lines
app/test-dma-perf/main.c new file +10 -0
+
+		token = strtok(NULL, ", ");
+	}
+
+	free(input);
+	return 0;
+}
+
+static int
+parse_lcore_dma(struct test_configure *test_case, const char

*value)

Quoted 40 lines

{

Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
app/test-dma-perf/main.c new file +24 -0
+	struct lcore_dma_map_t *lcore_dma_map;
+	char *input = strndup(value, strlen(value) + 1);
+	char *addrs = input;
+	char *ptrs[2];
+	char *start, *end, *substr;
+	uint16_t lcore_id;
+	int ret = 0;
+
+	while (*addrs == '\0')
+		addrs++;
+	if (*addrs == '\0') {
+		fprintf(stderr, "No input DMA addresses\n");
+		ret = -1;
+		goto out;
+	}
+
+	substr = strtok(addrs, ",");
+	if (substr == NULL) {
+		fprintf(stderr, "No input DMA address\n");
+		ret = -1;
+		goto out;
+	}
+
+	memset(&test_case->lcore_dma_map, 0, sizeof(struct

lcore_dma_map_t));

app/test-dma-perf/main.c new file +14 -0
+
+	do {
+		rte_strsplit(substr, strlen(substr), ptrs, 2, '@');
+
+		start = strstr(ptrs[0], "lcore");
+		if (start == NULL) {
+			fprintf(stderr, "Illegal lcore\n");
+			ret = -1;
+			break;
+		}
+
+		start += 5;
+		lcore_id = strtol(start, &end, 0);
+		if (end == start) {
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +1 -0
+			fprintf(stderr, "No input lcore ID or ID %d is

wrong\n", lcore_id);

app/test-dma-perf/main.c new file +6 -0
+			ret = -1;
+			break;
+		}
+
+		lcore_dma_map = &test_case->lcore_dma_map;
+		lcore_dma_map->lcores[lcore_dma_map->cnt] =

lcore_id;

Quoted 3 lines
Quoted 1 line
Quoted 1 line

app/test-dma-perf/main.c

Quoted 1 line
app/test-dma-perf/main.c new file +1 -0
+		strcpy(lcore_dma_map-

dma_names[lcore_dma_map-

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

cnt], ptrs[1]);

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

app/test-dma-perf/main.c

Quoted 11 lines
app/test-dma-perf/main.c new file +11 -0
+		lcore_dma_map->cnt++;
+		substr = strtok(NULL, ",");
+	} while (substr != NULL);
+
+out:
+	free(input);
+	return ret;
+}
+
+static int
+parse_entry(const char *value, struct test_configure_entry

*entry)

Quoted 316 lines
Quoted 316 lines

{

Quoted 54 lines
Quoted 54 lines
Quoted 54 lines
Quoted 54 lines
app/test-dma-perf/main.c new file +12 -0
+	char input[255] = {0};
+	char *args[MAX_PARAMS_PER_ENTRY];
+	int args_nr = -1;
+
+	if (value == NULL || entry == NULL)
+		goto out;
+
+	strncpy(input, value, 254);
+	if (*input == '\0')
+		goto out;
+
+	args_nr = rte_strsplit(input, strlen(input), args,

MAX_PARAMS_PER_ENTRY, ',');

app/test-dma-perf/main.c new file +35 -0
+	if (args_nr != 1 && args_nr != 4)
+		goto out;
+
+	entry->cur = entry->first = (uint32_t)atoi(args[0]);
+
+	if (args_nr == 4) {
+		entry->last = (uint32_t)atoi(args[1]);
+		entry->incr = (uint32_t)atoi(args[2]);
+		if (!strcmp(args[3], "MUL"))
+			entry->op = OP_MUL;
+		else if (!strcmp(args[3], "ADD"))
+			entry->op = OP_ADD;
+		else {
+			printf("Invalid op %s.\n", args[3]);
+			args_nr = -1;
+		}
+	} else {
+		entry->op = OP_NONE;
+		entry->last = 0;
+		entry->incr = 0;
+	}
+out:
+	return args_nr;
+}
+
+static uint16_t
+load_configs(const char *path)
+{
+	struct rte_cfgfile *cfgfile;
+	int nb_sections, i;
+	struct test_configure *test_case;
+	char section_name[CFG_NAME_LEN];
+	const char *case_type;
+	const char *lcore_dma;
+	const char *mem_size_str, *buf_size_str, *ring_size_str,

*kick_batch_str;

app/test-dma-perf/main.c new file +5 -0
+	int args_nr, nb_vp;
+	bool is_dma;
+
+	printf("config file parsing...\n");
+	cfgfile = rte_cfgfile_load(path, 0);
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +8 -0
+	if (!cfgfile) {
+		printf("Open configure file error.\n");
+		exit(1);
+	}
+
+	nb_sections = rte_cfgfile_num_sections(cfgfile, NULL, 0);
+	if (nb_sections > MAX_TEST_CASES) {
+		printf("Error: The maximum number of cases is

%d.\n",

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

app/test-dma-perf/main.c

Quoted 6 lines

MAX_TEST_CASES);

app/test-dma-perf/main.c new file +5 -0
+		exit(1);
+	}
+
+	for (i = 0; i < nb_sections; i++) {
+		snprintf(section_name, CFG_NAME_LEN, "case%d", i

+ 1);

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

app/test-dma-perf/main.c

Quoted 2 lines
app/test-dma-perf/main.c new file +2 -0
+		test_case = &test_cases[i];
+		case_type = rte_cfgfile_get_entry(cfgfile,

section_name,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/main.c

Quoted 3 lines

"type");

app/test-dma-perf/main.c new file +2 -0
+		if (!case_type) {
+			printf("Error: No case type in case %d, the

test will be

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

app/test-dma-perf/main.c

Quoted 10 lines

finished here.\n",

app/test-dma-perf/main.c new file +7 -0
+				i + 1);
+			test_case->is_valid = false;
+			continue;
+		}
+
+		if (strcmp(case_type, DMA_MEM_COPY) == 0) {
+			test_case->test_type =

TEST_TYPE_DMA_MEM_COPY;

app/test-dma-perf/main.c new file +1 -0
+			test_case->test_type_str =

DMA_MEM_COPY;

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

app/test-dma-perf/main.c

Quoted 5 lines
app/test-dma-perf/main.c new file +3 -0
+			is_dma = true;
+		} else if (strcmp(case_type, CPU_MEM_COPY) == 0) {
+			test_case->test_type =

TEST_TYPE_CPU_MEM_COPY;

app/test-dma-perf/main.c new file +1 -0
+			test_case->test_type_str =

CPU_MEM_COPY;

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/main.c

Quoted 3 lines
app/test-dma-perf/main.c new file +3 -0
+			is_dma = false;
+		} else {
+			printf("Error: Cannot find case type %s in

case%d.\n",

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

case_type, i + 1);

app/test-dma-perf/main.c new file +7 -0
+			test_case->is_valid = false;
+			continue;
+		}
+
+		nb_vp = 0;
+
+		test_case->src_numa_node =

(int)atoi(rte_cfgfile_get_entry(cfgfile,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines

+

	section_name, "src_numa_node"));
app/test-dma-perf/main.c new file +1 -0
+		test_case->dst_numa_node =

(int)atoi(rte_cfgfile_get_entry(cfgfile,
+

	section_name, "dst_numa_node"));
app/test-dma-perf/main.c new file +2 -0
+
+		mem_size_str = rte_cfgfile_get_entry(cfgfile,

section_name,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

app/test-dma-perf/main.c

Quoted 11 lines

"mem_size");

app/test-dma-perf/main.c new file +1 -0
+		args_nr = parse_entry(mem_size_str, &test_case-
Quoted 1 line

mem_size);

app/test-dma-perf/main.c new file +8 -0
+		if (args_nr < 0) {
+			printf("parse error in case %d.\n", i + 1);
+			test_case->is_valid = false;
+			continue;
+		} else if (args_nr > 1)
+			nb_vp++;
+
+		buf_size_str = rte_cfgfile_get_entry(cfgfile,

section_name,

Quoted 16 lines
Quoted 2 lines
Quoted 2 lines

app/test-dma-perf/main.c

Quoted 2 lines

"buf_size");

app/test-dma-perf/main.c new file +1 -0
+		args_nr = parse_entry(buf_size_str, &test_case-

buf_size);

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
app/test-dma-perf/main.c new file +5 -0
+		if (args_nr < 0) {
+			printf("parse error in case %d.\n", i + 1);
+			test_case->is_valid = false;
+			continue;
+		} else if (args_nr > 1)
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +4 -0
+			nb_vp++;
+
+		if (is_dma) {
+			ring_size_str = rte_cfgfile_get_entry(cfgfile,

section_name,
+

	"dma_ring_size");
app/test-dma-perf/main.c new file +1 -0
+			args_nr = parse_entry(ring_size_str,

&test_case-

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/main.c

Quoted 3 lines
Quoted 1 line

ring_size);

app/test-dma-perf/main.c new file +2 -0
+			if (args_nr < 0) {
+				printf("parse error in case %d.\n", i +

1);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

app/test-dma-perf/main.c

Quoted 6 lines
app/test-dma-perf/main.c new file +6 -0
+				test_case->is_valid = false;
+				continue;
+			} else if (args_nr > 1)
+				nb_vp++;
+
+			kick_batch_str =

rte_cfgfile_get_entry(cfgfile,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

app/test-dma-perf/main.c

Quoted 2 lines

section_name, "kick_batch");

app/test-dma-perf/main.c new file +1 -0
+			args_nr = parse_entry(kick_batch_str,

&test_case-

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/main.c

Quoted 3 lines
Quoted 1 line

kick_batch);

app/test-dma-perf/main.c new file +2 -0
+			if (args_nr < 0) {
+				printf("parse error in case %d.\n", i +

1);

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

app/test-dma-perf/main.c

Quoted 11 lines
app/test-dma-perf/main.c new file +6 -0
+				test_case->is_valid = false;
+				continue;
+			} else if (args_nr > 1)
+				nb_vp++;
+
+			lcore_dma = rte_cfgfile_get_entry(cfgfile,

section_name, "lcore_dma");

app/test-dma-perf/main.c new file +1 -0
+			int lcore_ret = parse_lcore_dma(test_case,

lcore_dma);

app/test-dma-perf/main.c new file +2 -0
+			if (lcore_ret < 0) {
+				printf("parse lcore dma error in case

%d.\n", i

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

1);

app/test-dma-perf/main.c

Quoted 7 lines
app/test-dma-perf/main.c new file +5 -0
+				test_case->is_valid = false;
+				continue;
+			}
+		} else {
+			lcore_dma = rte_cfgfile_get_entry(cfgfile,

section_name, "lcore");

app/test-dma-perf/main.c new file +1 -0
+			int lcore_ret = parse_lcore(test_case,

lcore_dma);

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

app/test-dma-perf/main.c

Quoted 2 lines
app/test-dma-perf/main.c new file +2 -0
+			if (lcore_ret < 0) {
+				printf("parse lcore error in case

%d.\n", i + 1);

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

app/test-dma-perf/main.c

Quoted 7 lines
app/test-dma-perf/main.c new file +7 -0
+				test_case->is_valid = false;
+				continue;
+			}
+		}
+
+		if (nb_vp > 1) {
+			printf("Error, each section can only have a

single

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

app/test-dma-perf/main.c

Quoted 7 lines

variable parameter.\n");

app/test-dma-perf/main.c new file +6 -0
+			test_case->is_valid = false;
+			continue;
+		}
+
+		test_case->cache_flush =
+			(int)atoi(rte_cfgfile_get_entry(cfgfile,

section_name,

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/main.c

Quoted 4 lines

"cache_flush"));

app/test-dma-perf/main.c new file +1 -0
+		test_case->test_secs =

(uint16_t)atoi(rte_cfgfile_get_entry(cfgfile,

app/test-dma-perf/main.c new file +1 -0
+					section_name,

"test_seconds"));

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
app/test-dma-perf/main.c new file +2 -0
+
+		test_case->eal_args = rte_cfgfile_get_entry(cfgfile,

section_name, "eal_args");

app/test-dma-perf/main.c new file +11 -0
+		test_case->is_valid = true;
+	}
+
+	rte_cfgfile_close(cfgfile);
+	printf("config file parsing complete.\n\n");
+	return i;
+}
+
+/* Parse the argument given in the command line of the
+application */ static int append_eal_args(int argc, char **argv,
+const char *eal_args, char **new_argv) {
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +8 -0
+	int i;
+	char *tokens[MAX_EAL_PARAM_NB];
+	char args[MAX_EAL_PARAM_LEN] = {0};
+	int token_nb, new_argc = 0;
+
+	for (i = 0; i < argc; i++) {
+		if ((strcmp(argv[i], CMDLINE_CONFIG_ARG) == 0) ||
+				(strcmp(argv[i],

CMDLINE_RESULT_ARG) ==

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

0)) {

app/test-dma-perf/main.c new file +4 -0
+			i++;
+			continue;
+		}
+		strlcpy(new_argv[new_argc], argv[i],
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines

sizeof(new_argv[new_argc]));

app/test-dma-perf/main.c new file +7 -0
+		new_argc++;
+	}
+
+	if (eal_args) {
+		strlcpy(args, eal_args, sizeof(args));
+		token_nb = rte_strsplit(args, strlen(args),
+					tokens,

MAX_EAL_PARAM_NB, ' ');

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
app/test-dma-perf/main.c new file +30 -0
+		for (i = 0; i < token_nb; i++)
+			strcpy(new_argv[new_argc++], tokens[i]);
+	}
+
+	return new_argc;
+}
+
+int
+main(int argc, char *argv[])
+{
+	int ret;
+	uint16_t case_nb;
+	uint32_t i, nb_lcores;
+	pid_t cpid, wpid;
+	int wstatus;
+	char args[MAX_EAL_PARAM_NB][MAX_EAL_PARAM_LEN];
+	char *pargs[MAX_EAL_PARAM_NB];
+	char *cfg_path_ptr = NULL;
+	char *rst_path_ptr = NULL;
+	char rst_path[PATH_MAX];
+	int new_argc;
+	bool is_first_case = true;
+
+	memset(args, 0, sizeof(args));
+
+	for (i = 0; i < RTE_DIM(pargs); i++)
+		pargs[i] = args[i];
+
+	for (i = 0; i < (uint32_t)argc; i++) {
+		if (strncmp(argv[i], CMDLINE_CONFIG_ARG,

MAX_LONG_OPT_SZ) == 0)

app/test-dma-perf/main.c new file +2 -0
+			cfg_path_ptr = argv[i + 1];
+		if (strncmp(argv[i], CMDLINE_RESULT_ARG,

MAX_LONG_OPT_SZ) == 0)

app/test-dma-perf/main.c new file +1 -0
+			rst_path_ptr = argv[i + 1];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +8 -0
+	}
+	if (cfg_path_ptr == NULL) {
+		printf("Config file not assigned.\n");
+		return -1;
+	}
+	if (rst_path_ptr == NULL) {
+		strlcpy(rst_path, cfg_path_ptr, PATH_MAX);
+		strcat(strtok(basename(rst_path), "."),

"_result.csv");

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
app/test-dma-perf/main.c new file +6 -0
+		rst_path_ptr = rst_path;
+	}
+
+	case_nb = load_configs(cfg_path_ptr);
+	fd = fopen(rst_path_ptr, "w");
+	if (fd == NULL) {
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +8 -0
+		printf("Open output CSV file error.\n");
+		return -1;
+	}
+	fclose(fd);
+
+	for (i = 0; i < case_nb; i++) {
+		if (test_cases[i].test_type == TEST_TYPE_NONE) {
+			printf("No test type in test case %d.\n\n", i +

1);

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +8 -0
+			continue;
+		}
+		if (!test_cases[i].is_valid) {
+			printf("Invalid test case %d.\n\n", i + 1);
+			continue;
+		}
+
+		cpid = fork();

[Anoob] Do we really need fork()? Can't we use code like,

Quoted 10 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
		RTE_LCORE_FOREACH_WORKER(lcore_id) {
			ret |= rte_eal_wait_lcore(lcore_id);
		}

to wait for all threads to exit?

[Cheng] Good question. Fork() is used here to establish a new

process

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

for the new test case. In order for each test case to have a new EAL environment (for the flexibility), the EAL must be reinitialized for

each

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

case.

Quoted 1 line
Quoted 1 line

However, the EAL parameters can only be initialized once per

process.

Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Therefore, we use a new process to run each new test case.

Moreover,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

each test case runs sequentially and does not affect the others, ensuring the accuracy of the performance data. Your code would

wait

Quoted 12 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

for all threads to exit in the same process. However, it would not

provide

Quoted 5 lines

a

Quoted 4 lines

"clean"

Quoted 3 lines

environment for each test case like fork() does. Fork() allows us to have a fully reinitialized environment, with no impact or side effects from previous test cases. This results in clean, precise performance

data

Quoted 4 lines
Quoted 4 lines

for

Quoted 3 lines

each case.

Quoted 2 lines

Please let me know your thoughts on this. And please let me know

if

Quoted 169 lines
Quoted 79 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

you have any other questions or require any clarification.

[Anoob] This was just a generic observation. I do not have a strong

opinion

Quoted 75 lines
Quoted 2 lines

either way.

[Cheng] sure, got it.

Quoted 15 lines
Quoted 15 lines

Thanks,
Cheng

Quoted 11 lines
Quoted 11 lines
app/test-dma-perf/main.c new file +7 -0
+		if (cpid < 0) {
+			printf("Fork case %d failed.\n", i + 1);
+			exit(EXIT_FAILURE);
+		} else if (cpid == 0) {
+			printf("\nRunning case %u\n\n", i + 1);
+
+			new_argc = append_eal_args(argc, argv,

test_cases[i].eal_args, pargs);

app/test-dma-perf/main.c new file +2 -0
+			ret = rte_eal_init(new_argc, pargs);
+			if (ret < 0)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +1 -0
+				rte_exit(EXIT_FAILURE, "Invalid EAL

arguments\n");

app/test-dma-perf/main.c new file +6 -0
+
+			/* Check lcores. */
+			nb_lcores = rte_lcore_count();
+			if (nb_lcores < 2)
+				rte_exit(EXIT_FAILURE,
+					"There should be at least 2

worker

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

app/test-dma-perf/main.c

Quoted 5 lines

lcores.\n");

app/test-dma-perf/main.c new file +4 -0
+
+			fd = fopen(rst_path_ptr, "a");
+			if (!fd) {
+				printf("Open output CSV file

error.\n");

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
app/test-dma-perf/main.c new file +18 -0
+				return 0;
+			}
+
+			if (is_first_case) {
+				output_env_info();
+				is_first_case = false;
+			}
+			run_test(i + 1, &test_cases[i]);
+
+			/* clean up the EAL */
+			rte_eal_cleanup();
+
+			fclose(fd);
+
+			printf("\nCase %u completed.\n\n", i + 1);
+
+			exit(EXIT_SUCCESS);
+		} else {
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.c

Quoted 8 lines
app/test-dma-perf/main.c new file +8 -0
+			wpid = waitpid(cpid, &wstatus, 0);
+			if (wpid == -1) {
+				printf("waitpid error.\n");
+				exit(EXIT_FAILURE);
+			}
+
+			if (WIFEXITED(wstatus))
+				printf("Case process exited. status

%d\n\n",

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/main.c

Quoted 3 lines
app/test-dma-perf/main.c new file +3 -0
+					WEXITSTATUS(wstatus));
+			else if (WIFSIGNALED(wstatus))
+				printf("Case process killed by signal

%d\n\n",

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

app/test-dma-perf/main.c

Quoted 3 lines
app/test-dma-perf/main.c new file +3 -0
+					WTERMSIG(wstatus));
+			else if (WIFSTOPPED(wstatus))
+				printf("Case process stopped by

signal

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/main.c

Quoted 4 lines

%d\n\n",

app/test-dma-perf/main.c new file +3 -0
+					WSTOPSIG(wstatus));
+			else if (WIFCONTINUED(wstatus))
+				printf("Case process

continued.\n\n");

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

app/test-dma-

Quoted 11 lines
app/test-dma-perf/main.c new file +2 -0
+			else
+				printf("Case process unknown

terminated.\n\n");

app/test-dma-perf/main.c new file +7 -0
+		}
+	}
+
+	printf("Bye...\n");
+	return 0;
+}
+
app/test-dma- renamed from app/test-dma-perf/main.h +0 -0

perf/main.h

Quoted 78 lines
Quoted 78 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

new file mode 100644 index 0000000000..215ac42673

app/test-dma-perf/main.h new file +15 -0
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Intel Corporation  */
+
+#ifndef _MAIN_H_
+#define _MAIN_H_
+
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_dev.h>
+#include <rte_dmadev.h>
+
+#ifndef __maybe_unused
+#define __maybe_unused	__rte_unused
+#endif
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.h line 23

Quoted 8 lines
app/test-dma-perf/main.h new file +8 -0
+
+#define MAX_WORKER_NB 128
+#define MAX_OUTPUT_STR_LEN 512
+
+#define MAX_DMA_NB 128
+#define MAX_LCORE_NB 256
+
+extern char

output_str[MAX_WORKER_NB][MAX_OUTPUT_STR_LEN];

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
app/test-dma-perf/main.h new file +10 -0
+
+typedef enum {
+	OP_NONE = 0,
+	OP_ADD,
+	OP_MUL
+} alg_op_type;
+
+struct test_configure_entry {
+	uint32_t first;
+	uint32_t last;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.h line 41

Quoted 8 lines
app/test-dma-perf/main.h new file +8 -0
+	uint32_t incr;
+	alg_op_type op;
+	uint32_t cur;
+};
+
+struct lcore_dma_map_t {
+	uint32_t lcores[MAX_WORKER_NB];
+	char

dma_names[MAX_WORKER_NB][RTE_DEV_NAME_MAX_LEN];

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
app/test-dma-perf/main.h new file +17 -0
+	int16_t dma_ids[MAX_WORKER_NB];
+	uint16_t cnt;
+};
+
+struct test_configure {
+	bool is_valid;
+	uint8_t test_type;
+	const char *test_type_str;
+	uint16_t src_numa_node;
+	uint16_t dst_numa_node;
+	uint16_t opcode;
+	bool is_dma;
+	struct lcore_dma_map_t lcore_dma_map;
+	struct test_configure_entry mem_size;
+	struct test_configure_entry buf_size;
+	struct test_configure_entry ring_size;
+	struct test_configure_entry kick_batch;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

app/test-dma-perf/main.h line 66

Quoted 8 lines
app/test-dma-perf/main.h new file +8 -0
+	uint32_t cache_flush;
+	uint32_t nr_buf;
+	uint16_t test_secs;
+	const char *eal_args;
+	uint8_t scenario_id;
+};
+
+void mem_copy_benchmark(struct test_configure *cfg, bool

is_dma);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
app/test-dma-perf/main.h new file +2 -0
+
+#endif /* _MAIN_H_ */
app/test-dma- renamed from app/test-dma-perf/meson.build +0 -0

perf/meson.build new file mode 100644 index

0000000000..bd6c264002

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

app/test-dma-perf/meson.build line 1

Quoted 4 lines
app/test-dma-perf/meson.build new file +1 -0
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2019-

2023

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
app/test-dma-perf/meson.build new file +16 -1
+Intel Corporation
+
+# meson file, for building this app as part of a main DPDK build.
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+deps += ['dmadev', 'mbuf', 'cfgfile']
+
+sources = files(
+        'main.c',
+        'benchmark.c',
+)
--

2.40.1

#24 lore raw

Hi Cheng,

Quoted 2 lines

[Cheng] sure, no problem. Feel free to contact me if you have any more questions.

No further comments from my side. Can you submit next version?

Top level, I think you may need to add a doc file for the new app.

Few issues that we are also checking in our end,
1. The app is having significant memory footprint. Need to see where we can improve.
2. EAL args passing doesn't seem to be working. Are you able to enable specific devices by making using of '-a' arg?

Thanks,
Anoob

#25 lore raw

Hi Cheng,

I think there is a bug in EAL parsing. Please check the below diff and see if we should do something similar.

app/test-dma-perf/main.c +1 -1
index d65655b87b..7fcaa5acf6 100644
@@ -432,7 +432,7 @@ append_eal_args(int argc, char **argv, const char *eal_args, char **new_argv)
                        i++;
                        continue;
                }
-               strlcpy(new_argv[new_argc], argv[i], sizeof(new_argv[new_argc]));
+               strlcpy(new_argv[new_argc], argv[i], MAX_EAL_PARAM_LEN);
                new_argc++;
        }
 

Thanks,
Anoob

Quoted 29 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 3:19 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

Quoted 2 lines

[Cheng] sure, no problem. Feel free to contact me if you have any more questions.

No further comments from my side. Can you submit next version?

Top level, I think you may need to add a doc file for the new app.

Few issues that we are also checking in our end, 1. The app is having significant memory footprint. Need to see where we can improve.
2. EAL args passing doesn't seem to be working. Are you able to enable specific devices by making using of '-a' arg?

Thanks,
Anoob

#26 lore raw

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 32 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 6:53 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

I think there is a bug in EAL parsing. Please check the below diff and see if we should do something similar.

app/test-dma-perf/main.c index renamed from app/test-dma-perf/main.c +0 -0

d65655b87b..7fcaa5acf6 100644

app/test-dma-perf/main.c +1 -1
@@ -432,7 +432,7 @@ append_eal_args(int argc, char **argv, const char
 *eal_args, char **new_argv)
                        i++;
                        continue;
                }
-               strlcpy(new_argv[new_argc], argv[i], sizeof(new_argv[new_argc]));
+               strlcpy(new_argv[new_argc], argv[i], MAX_EAL_PARAM_LEN);
                new_argc++;
        }
Quoted 2 lines

Thanks,
Anoob

[Cheng] yes there is an issue in it. And I have the same fix. I'll submit it later, thanks.

Quoted 10 lines
Quoted 10 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 3:19 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Quoted 11 lines

Wang,

Quoted 10 lines

YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

Quoted 2 lines

[Cheng] sure, no problem. Feel free to contact me if you have any more questions.

No further comments from my side. Can you submit next version?

[Cheng] Glad to know. Yes, the next version is almost ready, I'll submit it this weekend.

Quoted 2 lines
Quoted 2 lines

Top level, I think you may need to add a doc file for the new app.

[Cheng] sure, I was thinking about this, and I'll add a doc for this, thanks.

Quoted 5 lines
Quoted 5 lines

Few issues that we are also checking in our end, 1. The app is having significant memory footprint. Need to see where we can improve.
2. EAL args passing doesn't seem to be working. Are you able to enable specific devices by making using of '-a' arg?

[Cheng] I'll take a look at the significant memory footprint issue later, thanks. As for the eal args process, yes there is an issue, Dengdui's comments also mentioned it, and I have fixed it in the v7 patch.

Quoted 3 lines
Quoted 3 lines

Thanks,
Anoob

#27 lore raw

Hi Anoob,

I have a question about the doc.
Do you think I should add a folder in the doc/guides/ just like doc/guides/dma_perf?

Thanks,
Cheng

Quoted 106 lines

-----Original Message-----
From: Jiang, Cheng1
Sent: Friday, June 16, 2023 11:16 PM
To: Anoob Joseph <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 9 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 6:53 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 14 lines

YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

I think there is a bug in EAL parsing. Please check the below diff and see if we should do something similar.

app/test-dma-perf/main.c index renamed from app/test-dma-perf/main.c +0 -0

d65655b87b..7fcaa5acf6 100644

app/test-dma-perf/main.c append_eal_args(int argc, char **argv, const char · line 436

Quoted 5 lines
app/test-dma-perf/main.c +0 -1
@@ -432,7 +432,7 @@ append_eal_args(int argc, char **argv, const char
 *eal_args, char **new_argv)
                        i++;
                        continue;
                }
-               strlcpy(new_argv[new_argc], argv[i],

sizeof(new_argv[new_argc]));

Quoted 7 lines
app/test-dma-perf/main.c +2 -0
+               strlcpy(new_argv[new_argc], argv[i],
+ MAX_EAL_PARAM_LEN);
                new_argc++;
        }

Thanks,
Anoob

[Cheng] yes there is an issue in it. And I have the same fix. I'll submit it later, thanks.

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 3:19 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 4 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 1 line

YuanX <[email protected]>; He, Xingguang

<[email protected]>

Quoted 9 lines
Quoted 9 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

Quoted 2 lines

[Cheng] sure, no problem. Feel free to contact me if you have any more questions.

No further comments from my side. Can you submit next version?

[Cheng] Glad to know. Yes, the next version is almost ready, I'll submit it this weekend.

Quoted 2 lines
Quoted 2 lines

Top level, I think you may need to add a doc file for the new app.

[Cheng] sure, I was thinking about this, and I'll add a doc for this, thanks.

Quoted 5 lines
Quoted 5 lines

Few issues that we are also checking in our end, 1. The app is having significant memory footprint. Need to see where we can improve.
2. EAL args passing doesn't seem to be working. Are you able to enable specific devices by making using of '-a' arg?

[Cheng] I'll take a look at the significant memory footprint issue later, thanks. As for the eal args process, yes there is an issue, Dengdui's comments also mentioned it, and I have fixed it in the v7 patch.

Quoted 3 lines
Quoted 3 lines

Thanks,
Anoob

#28 lore raw

Hi Cheng,

Quoted 2 lines

Do you think I should add a folder in the doc/guides/ just like doc/guides/dma_perf?

I think it should be under doc/guides/tools. I'll let Thomas & DMA maintainers comment further.

You can use below patch as a reference,

commit 1f5cfe964eefe96b5f8de1fadf9cc8fd1e214240

Author: Anoob Joseph <[email protected]>
Date:   Thu Nov 3 18:16:11 2022 +0530

app/security-perf: add session performance test

Thanks,
Anoob

#29 lore raw

Hi Anoob,

Quoted 14 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Monday, June 19, 2023 1:49 PM
To: Jiang, Cheng1 <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; Laatz, Kevin <[email protected]>; Chengwen Feng <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Quoted 7 lines

Hi Cheng,

Quoted 2 lines

Do you think I should add a folder in the doc/guides/ just like doc/guides/dma_perf?

I think it should be under doc/guides/tools. I'll let Thomas & DMA maintainers comment further.

OK, I'll try to add the doc in the tools folder first.

Quoted 11 lines

You can use below patch as a reference,

commit 1f5cfe964eefe96b5f8de1fadf9cc8fd1e214240

Author: Anoob Joseph <[email protected]>
Date:   Thu Nov 3 18:16:11 2022 +0530

app/security-perf: add session performance test

Thanks,
Anoob

Sure, thanks a lot!
Cheng

#30 lore raw
In reply to Jiang, Cheng1

Hi Anoob,

I've looked into the memory footprint issue, and I didn't find any.
So could you please help to share the config file you are using? Maybe it can help me to accelerate the debug process.

Thanks a lot,
Cheng

Quoted 105 lines

-----Original Message-----
From: Jiang, Cheng1
Sent: Friday, June 16, 2023 11:16 PM
To: Anoob Joseph <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 9 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 6:53 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 24 lines

YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

I think there is a bug in EAL parsing. Please check the below diff and see if we should do something similar.

app/test-dma-perf/main.c index renamed from app/test-dma-perf/main.c +0 -0

d65655b87b..7fcaa5acf6 100644

app/test-dma-perf/main.c +2 -1
@@ -432,7 +432,7 @@ append_eal_args(int argc, char **argv, const char
 *eal_args, char **new_argv)
                        i++;
                        continue;
                }
-               strlcpy(new_argv[new_argc], argv[i], sizeof(new_argv[new_argc]));
+               strlcpy(new_argv[new_argc], argv[i],
+ MAX_EAL_PARAM_LEN);
                new_argc++;
        }
Quoted 2 lines

Thanks,
Anoob

[Cheng] yes there is an issue in it. And I have the same fix. I'll submit it later, thanks.

Quoted 8 lines
Quoted 8 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 3:19 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 4 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 1 line

YuanX <[email protected]>; He, Xingguang

<[email protected]>

Quoted 9 lines
Quoted 9 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

Quoted 2 lines

[Cheng] sure, no problem. Feel free to contact me if you have any more questions.

No further comments from my side. Can you submit next version?

[Cheng] Glad to know. Yes, the next version is almost ready, I'll submit it this weekend.

Quoted 2 lines
Quoted 2 lines

Top level, I think you may need to add a doc file for the new app.

[Cheng] sure, I was thinking about this, and I'll add a doc for this, thanks.

Quoted 5 lines
Quoted 5 lines

Few issues that we are also checking in our end, 1. The app is having significant memory footprint. Need to see where we can improve.
2. EAL args passing doesn't seem to be working. Are you able to enable specific devices by making using of '-a' arg?

[Cheng] I'll take a look at the significant memory footprint issue later, thanks. As for the eal args process, yes there is an issue, Dengdui's comments also mentioned it, and I have fixed it in the v7 patch.

Quoted 3 lines
Quoted 3 lines

Thanks,
Anoob

#31 lore raw

Hi Cheng,

Please see inline.

Thanks,
Anoob

Quoted 13 lines

-----Original Message-----
From: Jiang, Cheng1 <[email protected]>
Sent: Sunday, June 18, 2023 11:05 AM
To: Anoob Joseph <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Quoted 5 lines

Hi Anoob,

I've looked into the memory footprint issue, and I didn't find any. So could you please help to share the config file you are using? Maybe it can help me to accelerate the debug process.

[Anoob] We had to increase the hugepages to get the test running. Otherwise simple memory allocations were failing. Some of structs have redundant fields which can be easily addressed.

For example please check below members in struct lcore_params,
1. scenario_id
2. lcore_id
3. dma_name
4. worker_id
5. test_secs

Also, some of the parameters in the above struct is const for a thread (like buf_size, kick_batch etc). So I was thinking may be it is better to split lcore_params into two portions and have only the dynamic part as volatile. But that is something we can take up later. I leave that to your judgement.

Quoted 118 lines

Thanks a lot,
Cheng

Quoted 9 lines

-----Original Message-----
From: Jiang, Cheng1
Sent: Friday, June 16, 2023 11:16 PM
To: Anoob Joseph <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 1 line

YuanX <[email protected]>; He, Xingguang

<[email protected]>

Quoted 17 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 7 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 6:53 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 4 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 1 line

YuanX <[email protected]>; He, Xingguang

<[email protected]>

Quoted 8 lines
Quoted 8 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

I think there is a bug in EAL parsing. Please check the below diff and see if we should do something similar.

Quoted 11 lines

app/test-dma-perf/main.c append_eal_args(int argc, char **argv, const · line 436

Quoted 11 lines
app/test-dma-perf/main.c +0 -0

index
d65655b87b..7fcaa5acf6 100644

app/test-dma-perf/main.c +0 -1
@@ -432,7 +432,7 @@ append_eal_args(int argc, char **argv, const
 char *eal_args, char **new_argv)
                        i++;
                        continue;
                }
-               strlcpy(new_argv[new_argc], argv[i],

sizeof(new_argv[new_argc]));

Quoted 12 lines
Quoted 7 lines
app/test-dma-perf/main.c +2 -0
+               strlcpy(new_argv[new_argc], argv[i],
+ MAX_EAL_PARAM_LEN);
                new_argc++;
        }

Thanks,
Anoob

[Cheng] yes there is an issue in it. And I have the same fix. I'll submit it later, thanks.

Quoted 1 line
Quoted 10 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 3:19 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 2 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla
<[email protected]>; Ma, WenwuX

<[email protected]>;

Quoted 13 lines
Quoted 2 lines

Wang,

Quoted 1 line

YuanX <[email protected]>; He, Xingguang

<[email protected]>

Quoted 9 lines
Quoted 9 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

Quoted 2 lines

[Cheng] sure, no problem. Feel free to contact me if you have any more questions.

No further comments from my side. Can you submit next version?

Quoted 11 lines

[Cheng] Glad to know. Yes, the next version is almost ready, I'll submit it this weekend.

Quoted 2 lines
Quoted 2 lines

Top level, I think you may need to add a doc file for the new app.

[Cheng] sure, I was thinking about this, and I'll add a doc for this, thanks.

Quoted 3 lines
Quoted 3 lines

Few issues that we are also checking in our end, 1. The app is having significant memory footprint. Need to see where we can

improve.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

2. EAL args passing doesn't seem to be working. Are you able to enable specific devices by making using of '-a' arg?

[Cheng] I'll take a look at the significant memory footprint issue later,

thanks.

Quoted 6 lines

As for the eal args process, yes there is an issue, Dengdui's comments also mentioned it, and I have fixed it in the v7 patch.

Quoted 3 lines
Quoted 3 lines

Thanks,
Anoob

#32 lore raw

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 44 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Monday, June 19, 2023 1:26 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

Please see inline.

Thanks,
Anoob

Quoted 9 lines

-----Original Message-----
From: Jiang, Cheng1 <[email protected]>
Sent: Sunday, June 18, 2023 11:05 AM
To: Anoob Joseph <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 9 lines

YuanX <[email protected]>; He, Xingguang <[email protected]>
Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Anoob,

I've looked into the memory footprint issue, and I didn't find any. So could you please help to share the config file you are using? Maybe it can help me to accelerate the debug process.

[Anoob] We had to increase the hugepages to get the test running. Otherwise simple memory allocations were failing. Some of structs have redundant fields which can be easily addressed.

Quoted 8 lines

For example please check below members in struct lcore_params, 1. scenario_id 2. lcore_id 3. dma_name 4. worker_id 5. test_secs

Also, some of the parameters in the above struct is const for a thread (like buf_size, kick_batch etc). So I was thinking may be it is better to split lcore_params into two portions and have only the dynamic part as volatile. But that is something we can take up later. I leave that to your judgement.

[Cheng] OK, I'll take a look and try to make some improvements. Thanks for your advice.

Quoted 120 lines
Quoted 11 lines

Thanks a lot,
Cheng

Quoted 7 lines

-----Original Message-----
From: Jiang, Cheng1
Sent: Friday, June 16, 2023 11:16 PM
To: Anoob Joseph <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 108 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 1 line

YuanX <[email protected]>; He, Xingguang

<[email protected]>

Quoted 17 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 7 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 6:53 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 4 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla <[email protected]>; Ma, WenwuX <[email protected]>;

Wang,

Quoted 1 line

YuanX <[email protected]>; He, Xingguang

<[email protected]>

Quoted 8 lines
Quoted 8 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

I think there is a bug in EAL parsing. Please check the below diff and see if we should do something similar.

Quoted 11 lines

app/test-dma-perf/main.c append_eal_args(int argc, char **argv, const · line 436

Quoted 11 lines
app/test-dma-perf/main.c +0 -0

index
d65655b87b..7fcaa5acf6 100644

app/test-dma-perf/main.c +0 -1
@@ -432,7 +432,7 @@ append_eal_args(int argc, char **argv, const
 char *eal_args, char **new_argv)
                        i++;
                        continue;
                }
-               strlcpy(new_argv[new_argc], argv[i],

sizeof(new_argv[new_argc]));

Quoted 12 lines
Quoted 7 lines
app/test-dma-perf/main.c +2 -0
+               strlcpy(new_argv[new_argc], argv[i],
+ MAX_EAL_PARAM_LEN);
                new_argc++;
        }

Thanks,
Anoob

[Cheng] yes there is an issue in it. And I have the same fix. I'll submit it later, thanks.

Quoted 1 line
Quoted 10 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 16, 2023 3:19 PM
To: Jiang, Cheng1 <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia,

Chenbo

Quoted 2 lines
Quoted 2 lines

<[email protected]>; Amit Prakash Shukla
<[email protected]>; Ma, WenwuX

<[email protected]>;

Quoted 13 lines
Quoted 2 lines

Wang,

Quoted 1 line

YuanX <[email protected]>; He, Xingguang

<[email protected]>

Quoted 9 lines
Quoted 9 lines

Subject: RE: [EXT] [PATCH v6] app/dma-perf: introduce dma-perf application

Hi Cheng,

Quoted 2 lines

[Cheng] sure, no problem. Feel free to contact me if you have any more questions.

No further comments from my side. Can you submit next version?

Quoted 11 lines

[Cheng] Glad to know. Yes, the next version is almost ready, I'll submit it this weekend.

Quoted 2 lines
Quoted 2 lines

Top level, I think you may need to add a doc file for the new app.

[Cheng] sure, I was thinking about this, and I'll add a doc for this, thanks.

Quoted 3 lines
Quoted 3 lines

Few issues that we are also checking in our end, 1. The app is having significant memory footprint. Need to see where we can

improve.

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

2. EAL args passing doesn't seem to be working. Are you able to enable specific devices by making using of '-a' arg?

[Cheng] I'll take a look at the significant memory footprint issue later,

thanks.

Quoted 6 lines

As for the eal args process, yes there is an issue, Dengdui's comments also mentioned it, and I have fixed it in the v7 patch.

Quoted 3 lines
Quoted 3 lines

Thanks,
Anoob

#33 lore raw

PATCH v7 1/1 app/dma-perf: introduce dma-perf application

In reply to Cheng Jiang

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are supported for now.

---
v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:

 app/meson.build               |   1 +
 app/test-dma-perf/benchmark.c | 498 ++++++++++++++++++++++++++++
 app/test-dma-perf/config.ini  |  61 ++++
 app/test-dma-perf/main.c      | 594 ++++++++++++++++++++++++++++++++++
 app/test-dma-perf/main.h      |  69 ++++
 app/test-dma-perf/meson.build |  17 +
 6 files changed, 1240 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c
 create mode 100644 app/test-dma-perf/config.ini
 create mode 100644 app/test-dma-perf/main.c
 create mode 100644 app/test-dma-perf/main.h
 create mode 100644 app/test-dma-perf/meson.build
app/meson.build +1 -0
index 74d2420f67..4fc1a83eba 100644
@@ -19,6 +19,7 @@ apps = [
         'test-cmdline',
         'test-compress-perf',
         'test-crypto-perf',
+        'test-dma-perf',
         'test-eventdev',
         'test-fib',
         'test-flow-perf',
app/test-dma-perf/benchmark.c new file +498 -0

498 lines not on this page · raw message

app/test-dma-perf/config.ini new file +61 -0

61 lines not on this page · raw message

app/test-dma-perf/main.c new file +594 -0

594 lines not on this page · raw message

app/test-dma-perf/main.h new file +69 -0

69 lines not on this page · raw message

app/test-dma-perf/meson.build new file +17 -0
new file mode 100644
index 0000000000..bd6c264002
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2023 Intel Corporation
+
+# meson file, for building this app as part of a main DPDK build.
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+deps += ['dmadev', 'mbuf', 'cfgfile']
+
+sources = files(
+        'main.c',
+        'benchmark.c',
+)

--
2.40.1

#34 lore raw

PATCH v8 1/1 app/dma-perf: introduce dma-perf application

In reply to Cheng Jiang

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are supported for now.

---
v8:
fixed string copy issue in parse_lcore();
improved some data display format;
added doc in doc/guides/tools;
updated release notes;

v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 498 +++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 594 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  69 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1350 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c
 create mode 100644 app/test-dma-perf/config.ini
 create mode 100644 app/test-dma-perf/main.c
 create mode 100644 app/test-dma-perf/main.h
 create mode 100644 app/test-dma-perf/meson.build
 create mode 100644 doc/guides/tools/dmaperf.rst
app/meson.build +1 -0
index 74d2420f67..4fc1a83eba 100644
@@ -19,6 +19,7 @@ apps = [
         'test-cmdline',
         'test-compress-perf',
         'test-crypto-perf',
+        'test-dma-perf',
         'test-eventdev',
         'test-fib',
         'test-flow-perf',
app/test-dma-perf/benchmark.c new file +498 -0

498 lines not on this page · raw message

app/test-dma-perf/config.ini new file +61 -0

61 lines not on this page · raw message

app/test-dma-perf/main.c new file +594 -0

594 lines not on this page · raw message

app/test-dma-perf/main.h new file +69 -0

69 lines not on this page · raw message

app/test-dma-perf/meson.build new file +17 -0
new file mode 100644
index 0000000000..bd6c264002
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2023 Intel Corporation
+
+# meson file, for building this app as part of a main DPDK build.
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+deps += ['dmadev', 'mbuf', 'cfgfile']
+
+sources = files(
+        'main.c',
+        'benchmark.c',
+)
doc/guides/rel_notes/release_23_07.rst +6 -0
index 027ae7bd2d..2355b558bc 100644
@@ -170,6 +170,12 @@ New Features
 
   See :doc:`../prog_guide/pdcp_lib` for more information.
 
+* **Added DMA device performance test application.**
+
+  Added an new application to test the performance of DMA device and CPU.
+
+  See the :doc:`../tools/dmaperf` for more details.
+
 
 Removed Items
 -------------
doc/guides/tools/dmaperf.rst new file +103 -0

103 lines not on this page · raw message

doc/guides/tools/index.rst +1 -0
index 6f84fc31ff..857572da96 100644
@@ -23,3 +23,4 @@ DPDK Tools User Guides
     testregex
     testmldev
     dts
+    dmaperf

--
2.40.1

#35 lore raw

Hi Cheng,

Thanks for the new version. Please see inline.

Thanks,
Anoob

Quoted 78 lines

-----Original Message-----
From: Cheng Jiang <[email protected]>
Sent: Tuesday, June 20, 2023 12:24 PM
To: [email protected]; [email protected]; [email protected]; [email protected]; Amit Prakash Shukla <[email protected]>; Anoob Joseph <[email protected]>; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Cheng Jiang <[email protected]>
Subject: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

External Email

----------------------------------------------------------------------
There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre- set config file. Memory Copy performance test are supported for now.

---
v8:
fixed string copy issue in parse_lcore();
improved some data display format;
added doc in doc/guides/tools;
updated release notes;

v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 498 +++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 594 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  69 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1350 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c  create mode 100644

app/test-dma-perf/config.ini create mode 100644 app/test-dma- perf/main.c create mode 100644 app/test-dma-perf/main.h create mode 100644 app/test-dma-perf/meson.build create mode 100644 doc/guides/tools/dmaperf.rst

<snip>

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +8 -0
+/* Configuration of device. */
+static void
+configure_dmadev_queue(uint32_t dev_id, uint32_t ring_size) {
+	uint16_t vchan = 0;
+	struct rte_dma_info info;
+	struct rte_dma_conf dev_config = { .nb_vchans = 1 };
+	struct rte_dma_vchan_conf qconf = {
+		.direction = RTE_DMA_DIR_MEM_TO_MEM,

app/test-dma-perf/benchmark.c line 150

Quoted 10 lines
app/test-dma-perf/benchmark.c new file +10 -0
+		.nb_desc = ring_size
+	};
+
+	if (rte_dma_configure(dev_id, &dev_config) != 0)
+		rte_exit(EXIT_FAILURE, "Error with dma configure.\n");
+
+	if (rte_dma_vchan_setup(dev_id, vchan, &qconf) != 0)
+		rte_exit(EXIT_FAILURE, "Error with queue configuration.\n");
+
+	rte_dma_info_get(dev_id, &info);

[Anoob] This API can return errors. Better to add handling.

Quoted 12 lines
(quoted diff) +2 -0
+	if (info.nb_vchans != 1)
+		rte_exit(EXIT_FAILURE, "Error, no configured queues

reported on device id. %u\n",

app/test-dma-perf/benchmark.c new file +9 -0
+				dev_id);
+
+	if (rte_dma_start(dev_id) != 0)
+		rte_exit(EXIT_FAILURE, "Error with dma start.\n"); }
+
+static int
+config_dmadevs(struct test_configure *cfg) {
+	uint32_t ring_size = cfg->ring_size.cur;
+	struct lcore_dma_map_t *ldm = &cfg->lcore_dma_map;

app/test-dma-perf/benchmark.c line 173

Quoted 10 lines
app/test-dma-perf/benchmark.c new file +10 -0
+	uint32_t nb_workers = ldm->cnt;
+	uint32_t i;
+	int dev_id;
+	uint16_t nb_dmadevs = 0;
+	char *dma_name;
+
+	for (i = 0; i < ldm->cnt; i++) {
+		dma_name = ldm->dma_names[i];
+		dev_id = rte_dma_get_dev_id_by_name(dma_name);
+		if (dev_id == -1) {

[Anoob] Can you check the above API definition? I think it returns not just -1 in case of errors.

Quoted 37 lines
(quoted diff) +1 -0
+			fprintf(stderr, "Error: Fail to find DMA %s.\n",

dma_name);

app/test-dma-perf/benchmark.c new file +11 -0
+			goto end;
+		}
+
+		ldm->dma_ids[i] = dev_id;
+		configure_dmadev_queue(dev_id, ring_size);
+		++nb_dmadevs;
+	}
+
+end:
+	if (nb_dmadevs < nb_workers) {
+		printf("Not enough dmadevs (%u) for all workers (%u).\n",

nb_dmadevs, nb_workers);

app/test-dma-perf/benchmark.c new file +22 -0
+		return -1;
+	}
+
+	printf("Number of used dmadevs: %u.\n", nb_dmadevs);
+
+	return 0;
+}
+
+static inline void
+do_dma_submit_and_poll(uint16_t dev_id, uint64_t *async_cnt,
+			volatile struct worker_info *worker_info) {
+	int ret;
+	uint16_t nr_cpl;
+
+	ret = rte_dma_submit(dev_id, 0);
+	if (ret < 0) {
+		rte_dma_stop(dev_id);
+		rte_dma_close(dev_id);
+		rte_exit(EXIT_FAILURE, "Error with dma submit.\n");
+	}
+
+	nr_cpl = rte_dma_completed(dev_id, 0, MAX_DMA_CPL_NB, NULL,

NULL);

app/test-dma-perf/benchmark.c line 214

Quoted 6 lines
app/test-dma-perf/benchmark.c new file +6 -0
+	*async_cnt -= nr_cpl;
+	worker_info->total_cpl += nr_cpl;
+}
+
+static inline int
+do_dma_mem_copy(void *p)

[Anoob] Just curious, why not pass struct lcore_params *para itself? Is it because the pointer is volatile? If yes, then we can take an AI to split the struct into volatile and non-volatile parts.

Quoted 27 lines
app/test-dma-perf/benchmark.c new file +26 -0
+{
+	const uint16_t *para_idx = (uint16_t *)p;
+	volatile struct lcore_params *para = lcores_p[*para_idx].v_ptr;
+	volatile struct worker_info *worker_info = &(para->worker_info);
+	const uint16_t dev_id = para->dev_id;
+	const uint32_t nr_buf = para->nr_buf;
+	const uint16_t kick_batch = para->kick_batch;
+	const uint32_t buf_size = para->buf_size;
+	struct rte_mbuf **srcs = para->srcs;
+	struct rte_mbuf **dsts = para->dsts;
+	uint16_t nr_cpl;
+	uint64_t async_cnt = 0;
+	uint32_t i;
+	uint32_t poll_cnt = 0;
+	int ret;
+
+	worker_info->stop_flag = false;
+	worker_info->ready_flag = true;
+
+	while (!worker_info->start_flag)
+		;
+
+	while (1) {
+		for (i = 0; i < nr_buf; i++) {
+dma_copy:
+			ret = rte_dma_copy(dev_id, 0,

rte_pktmbuf_iova(srcs[i]),

app/test-dma-perf/benchmark.c line 241

Quoted 1 line
app/test-dma-perf/benchmark.c new file +1 -0
+				rte_pktmbuf_iova(dsts[i]), buf_size, 0);

[Anoob] Do we need to use ' rte_mbuf_data_iova' here instead of 'rte_pktmbuf_iova'?

app/test-dma-perf/benchmark.c line 248

Quoted 8 lines
app/test-dma-perf/benchmark.c new file +3 -0
+			if (unlikely(ret < 0)) {
+				if (ret == -ENOSPC) {
+					do_dma_submit_and_poll(dev_id,

&async_cnt, worker_info);

app/test-dma-perf/benchmark.c new file +4 -0
+					goto dma_copy;
+				} else {
+					/* Error exit */
+					rte_dma_stop(dev_id);

[Anoob] Missing rte_dma_close() here. Also, may be introduce a static void function so that rte_exit call etc won't be part of the fastpath loop.

May be something like below and you can call it from here and "do_dma_submit_and_poll".

static void
error_exit(int dev_id)

{
	/* Error exit */
	rte_dma_stop(dev_id);
	rte_dma_close(dev_id);
	rte_exit(EXIT_FAILURE, "DMA enqueue failed\n");
}
Quoted 25 lines
(quoted diff) +1 -0
+					rte_exit(EXIT_FAILURE, "DMA

enqueue failed\n");

app/test-dma-perf/benchmark.c new file +6 -0
+				}
+			}
+			async_cnt++;
+
+			if ((async_cnt % kick_batch) == 0)
+				do_dma_submit_and_poll(dev_id,

&async_cnt, worker_info);

app/test-dma-perf/benchmark.c new file +9 -0
+		}
+
+		if (worker_info->stop_flag)
+			break;
+	}
+
+	rte_dma_submit(dev_id, 0);
+	while ((async_cnt > 0) && (poll_cnt++ < POLL_MAX)) {
+		nr_cpl = rte_dma_completed(dev_id, 0,

MAX_DMA_CPL_NB, NULL, NULL);

app/test-dma-perf/benchmark.c new file +6 -0
+		async_cnt -= nr_cpl;
+	}
+
+	return 0;
+}
+

<snip>

Quoted 41 lines
app/test-dma-perf/benchmark.c new file +12 -0
+static int
+setup_memory_env(struct test_configure *cfg, struct rte_mbuf ***srcs,
+			struct rte_mbuf ***dsts)
+{
+	unsigned int buf_size = cfg->buf_size.cur;
+	unsigned int nr_sockets;
+	uint32_t nr_buf = cfg->nr_buf;
+
+	nr_sockets = rte_socket_count();
+	if (cfg->src_numa_node >= nr_sockets ||
+		cfg->dst_numa_node >= nr_sockets) {
+		printf("Error: Source or destination numa exceeds the acture

numa nodes.\n");

app/test-dma-perf/benchmark.c new file +28 -0
+		return -1;
+	}
+
+	src_pool = rte_pktmbuf_pool_create("Benchmark_DMA_SRC",
+			nr_buf,
+			0,
+			0,
+			buf_size + RTE_PKTMBUF_HEADROOM,
+			cfg->src_numa_node);
+	if (src_pool == NULL) {
+		PRINT_ERR("Error with source mempool creation.\n");
+		return -1;
+	}
+
+	dst_pool = rte_pktmbuf_pool_create("Benchmark_DMA_DST",
+			nr_buf,
+			0,
+			0,
+			buf_size + RTE_PKTMBUF_HEADROOM,
+			cfg->dst_numa_node);
+	if (dst_pool == NULL) {
+		PRINT_ERR("Error with destination mempool creation.\n");
+		return -1;
+	}
+
+	*srcs = rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *), 0);
+	if (*srcs == NULL) {
+		printf("Error: srcs malloc failed.\n");

app/test-dma-perf/benchmark.c line 353

Quoted 10 lines
app/test-dma-perf/benchmark.c new file +10 -0
+		return -1;
+	}
+
+	*dsts = rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *), 0);
+	if (*dsts == NULL) {
+		printf("Error: dsts malloc failed.\n");
+		return -1;
+	}
+
+	if (rte_mempool_get_bulk(src_pool, (void **)*srcs, nr_buf) != 0) {

[Anoob] Might be better to use 'rte_pktmbuf_alloc_bulk' since we use ' rte_mbuf_data_iova' in the datapath and it is desirable to initialize it properly as an mbuf. Same comment below as well.

<snip>

Review

Quoted 11 lines
app/test-dma-perf/benchmark.c new file +7 -0
+
+	for (i = 0; i < nb_workers; i++) {
+		calc_result(buf_size, nr_buf, nb_workers, test_secs,
+			lcores_p[i].v_ptr->worker_info.test_cpl,
+			&memory, &avg_cycles, &bandwidth, &mops);
+		output_result(cfg->scenario_id, lcores_p[i].v_ptr->lcore_id,
+					lcores_p[i].v_ptr->dma_name,

avg_cycles, buf_size,

(quoted diff) +1 -0
+					nr_buf / nb_workers, memory,

bandwidth, mops, is_dma);

(quoted diff) +1 -0
+	}

[Anoob] Can you also print total_bandwidth & total_mops? It can be a simple aggregation in the above loop. Would help when we are dealing with larger number of queues but single hardware block.

app/test-dma-perf/benchmark.c line 466

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+
+out:
+	/* free mbufs used in the test */
+	if (srcs)

[Anoob] DPDK coding guidelines recommend a usage like below,

	(if (srcs != NULL)
Quoted 7 lines
app/test-dma-perf/benchmark.c new file +7 -0
+		rte_pktmbuf_free_bulk(srcs, nr_buf);
+	if (dsts)
+		rte_pktmbuf_free_bulk(dsts, nr_buf);
+
+	/* free the points for the mbufs */
+	rte_free(srcs);
+	srcs = NULL;

app/test-dma-perf/benchmark.c line 483

Quoted 10 lines
app/test-dma-perf/benchmark.c new file +10 -0
+	rte_free(dsts);
+	dsts = NULL;
+
+	if (src_pool) {
+		rte_mempool_free(src_pool);
+		src_pool = NULL;
+	}
+	if (dst_pool) {
+		rte_mempool_free(dst_pool);
+		src_pool = NULL;

[Anoob] Should be dst_pool, right?

<snip>

Quoted 7 lines
app/test-dma-perf/main.h new file renamed from app/test-dma-perf/main.h +0 -0

mode 100644 index 0000000000..215ac42673

app/test-dma-perf/main.h new file +2 -0
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Intel Corporation

app/test-dma-perf/main.h line 12

Quoted 10 lines
app/test-dma-perf/main.h new file +10 -0
+ */
+
+#ifndef _MAIN_H_
+#define _MAIN_H_
+
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_dev.h>
+#include <rte_dmadev.h>

[Anoob] Is the above include (rte_dmadev.h) required?

app/test-dma-perf/main.h line 16

Quoted 4 lines
app/test-dma-perf/main.h new file +4 -0
+
+#ifndef __maybe_unused
+#define __maybe_unused	__rte_unused
+#endif

[Anoob] Can you try to avoid this and use rte_unused or RTE_SET_USED in the cache_flush_buf() function?

#36 lore raw

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 117 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 23, 2023 2:53 PM
To: Jiang, Cheng1 <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; [email protected]; Laatz, Kevin <[email protected]>; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>
Subject: RE: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

Hi Cheng,

Thanks for the new version. Please see inline.

Thanks,
Anoob

Quoted 14 lines

-----Original Message-----
From: Cheng Jiang <[email protected]>
Sent: Tuesday, June 20, 2023 12:24 PM
To: [email protected]; [email protected]; [email protected]; [email protected]; Amit Prakash Shukla <[email protected]>; Anoob Joseph <[email protected]>; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Cheng Jiang <[email protected]>
Subject: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

Quoted 4 lines

External Email

----------------------------------------------------------------------
There are many high-performance DMA devices supported in DPDK now,

and

Quoted 9 lines

these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre- set config file. Memory Copy performance test

are supported for now.

Quoted 51 lines

---
v8:
fixed string copy issue in parse_lcore();
improved some data display format;
added doc in doc/guides/tools;
updated release notes;

v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 498 +++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 594 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  69 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1350 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c  create mode 100644

app/test-dma-perf/config.ini create mode 100644 app/test-dma- perf/main.c create mode 100644 app/test-dma-perf/main.h create mode 100644 app/test-dma-perf/meson.build create mode 100644 doc/guides/tools/dmaperf.rst

<snip>

Quoted 12 lines
app/test-dma-perf/benchmark.c new file +12 -0
+/* Configuration of device. */
+static void
+configure_dmadev_queue(uint32_t dev_id, uint32_t ring_size) {
+	uint16_t vchan = 0;
+	struct rte_dma_info info;
+	struct rte_dma_conf dev_config = { .nb_vchans = 1 };
+	struct rte_dma_vchan_conf qconf = {
+		.direction = RTE_DMA_DIR_MEM_TO_MEM,
+		.nb_desc = ring_size
+	};
+
+	if (rte_dma_configure(dev_id, &dev_config) != 0)
Quoted 8 lines

app/test-dma-perf/benchmark.c line 150

Quoted 6 lines
app/test-dma-perf/benchmark.c new file +6 -0
+		rte_exit(EXIT_FAILURE, "Error with dma configure.\n");
+
+	if (rte_dma_vchan_setup(dev_id, vchan, &qconf) != 0)
+		rte_exit(EXIT_FAILURE, "Error with queue configuration.\n");
+
+	rte_dma_info_get(dev_id, &info);

[Anoob] This API can return errors. Better to add handling.

[Cheng] Sure, I'll fix it in the next version.

Quoted 18 lines
Quoted 18 lines
(quoted diff) +2 -0
+	if (info.nb_vchans != 1)
+		rte_exit(EXIT_FAILURE, "Error, no configured queues

reported on device id. %u\n",

app/test-dma-perf/benchmark.c new file +14 -0
+				dev_id);
+
+	if (rte_dma_start(dev_id) != 0)
+		rte_exit(EXIT_FAILURE, "Error with dma start.\n"); }
+
+static int
+config_dmadevs(struct test_configure *cfg) {
+	uint32_t ring_size = cfg->ring_size.cur;
+	struct lcore_dma_map_t *ldm = &cfg->lcore_dma_map;
+	uint32_t nb_workers = ldm->cnt;
+	uint32_t i;
+	int dev_id;
+	uint16_t nb_dmadevs = 0;
+	char *dma_name;
Quoted 8 lines

app/test-dma-perf/benchmark.c line 173

Quoted 5 lines
app/test-dma-perf/benchmark.c new file +5 -0
+
+	for (i = 0; i < ldm->cnt; i++) {
+		dma_name = ldm->dma_names[i];
+		dev_id = rte_dma_get_dev_id_by_name(dma_name);
+		if (dev_id == -1) {

[Anoob] Can you check the above API definition? I think it returns not just -1 in case of errors.

[Cheng] Yes, you are right, I'll fix it in the next version. Thanks a lot.

Quoted 40 lines
Quoted 40 lines
(quoted diff) +1 -0
+			fprintf(stderr, "Error: Fail to find DMA %s.\n",

dma_name);

app/test-dma-perf/benchmark.c new file +11 -0
+			goto end;
+		}
+
+		ldm->dma_ids[i] = dev_id;
+		configure_dmadev_queue(dev_id, ring_size);
+		++nb_dmadevs;
+	}
+
+end:
+	if (nb_dmadevs < nb_workers) {
+		printf("Not enough dmadevs (%u) for all workers (%u).\n",

nb_dmadevs, nb_workers);

app/test-dma-perf/benchmark.c new file +22 -0
+		return -1;
+	}
+
+	printf("Number of used dmadevs: %u.\n", nb_dmadevs);
+
+	return 0;
+}
+
+static inline void
+do_dma_submit_and_poll(uint16_t dev_id, uint64_t *async_cnt,
+			volatile struct worker_info *worker_info) {
+	int ret;
+	uint16_t nr_cpl;
+
+	ret = rte_dma_submit(dev_id, 0);
+	if (ret < 0) {
+		rte_dma_stop(dev_id);
+		rte_dma_close(dev_id);
+		rte_exit(EXIT_FAILURE, "Error with dma submit.\n");
+	}
+
+	nr_cpl = rte_dma_completed(dev_id, 0, MAX_DMA_CPL_NB, NULL,

NULL);

app/test-dma-perf/benchmark.c new file +2 -0
+	*async_cnt -= nr_cpl;
+	worker_info->total_cpl += nr_cpl;
Quoted 8 lines

app/test-dma-perf/benchmark.c line 214

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+}
+
+static inline int
+do_dma_mem_copy(void *p)

[Anoob] Just curious, why not pass struct lcore_params *para itself? Is it because the pointer is volatile? If yes, then we can take an AI to split the struct into volatile and non-volatile parts.

[Cheng] The reason I did it this way is because I want to launch this function on another core by spawning a new thread, and rte_eal_remote_launch() takes a void * as the parameter. That's why I passed void *p. Your suggestion to split the struct into volatile and non-volatile parts is quite reasonable. I am thinking about the best way to implement it. Thanks.

Quoted 24 lines
Quoted 24 lines
app/test-dma-perf/benchmark.c new file +23 -0
+{
+	const uint16_t *para_idx = (uint16_t *)p;
+	volatile struct lcore_params *para = lcores_p[*para_idx].v_ptr;
+	volatile struct worker_info *worker_info = &(para->worker_info);
+	const uint16_t dev_id = para->dev_id;
+	const uint32_t nr_buf = para->nr_buf;
+	const uint16_t kick_batch = para->kick_batch;
+	const uint32_t buf_size = para->buf_size;
+	struct rte_mbuf **srcs = para->srcs;
+	struct rte_mbuf **dsts = para->dsts;
+	uint16_t nr_cpl;
+	uint64_t async_cnt = 0;
+	uint32_t i;
+	uint32_t poll_cnt = 0;
+	int ret;
+
+	worker_info->stop_flag = false;
+	worker_info->ready_flag = true;
+
+	while (!worker_info->start_flag)
+		;
+
+	while (1) {
Quoted 8 lines

app/test-dma-perf/benchmark.c line 241

Quoted 5 lines
app/test-dma-perf/benchmark.c new file +3 -0
+		for (i = 0; i < nr_buf; i++) {
+dma_copy:
+			ret = rte_dma_copy(dev_id, 0,

rte_pktmbuf_iova(srcs[i]),

app/test-dma-perf/benchmark.c new file +1 -0
+				rte_pktmbuf_iova(dsts[i]), buf_size, 0);

[Anoob] Do we need to use ' rte_mbuf_data_iova' here instead of 'rte_pktmbuf_iova'?

[Cheng] yes rte_mbuf_data_iova is more appropriate, I'll fix it in the next version. Thanks.

Quoted 16 lines

app/test-dma-perf/benchmark.c line 248

Quoted 9 lines
app/test-dma-perf/benchmark.c new file +3 -0
+			if (unlikely(ret < 0)) {
+				if (ret == -ENOSPC) {
+					do_dma_submit_and_poll(dev_id,

&async_cnt, worker_info);

app/test-dma-perf/benchmark.c new file +4 -0
+					goto dma_copy;
+				} else {
+					/* Error exit */
+					rte_dma_stop(dev_id);

[Anoob] Missing rte_dma_close() here. Also, may be introduce a static void function so that rte_exit call etc won't be part of the fastpath loop.

May be something like below and you can call it from here and "do_dma_submit_and_poll".

Quoted 8 lines

static void
error_exit(int dev_id)

{
	/* Error exit */
	rte_dma_stop(dev_id);
	rte_dma_close(dev_id);
	rte_exit(EXIT_FAILURE, "DMA enqueue failed\n"); }

[Cheng] I'm not so sure here. rte_dma_close() is called in the rte_exit(). Do we still call it explicitly before rte_exit()?

Quoted 75 lines
Quoted 25 lines
(quoted diff) +1 -0
+					rte_exit(EXIT_FAILURE, "DMA

enqueue failed\n");

app/test-dma-perf/benchmark.c new file +6 -0
+				}
+			}
+			async_cnt++;
+
+			if ((async_cnt % kick_batch) == 0)
+				do_dma_submit_and_poll(dev_id,

&async_cnt, worker_info);

app/test-dma-perf/benchmark.c new file +9 -0
+		}
+
+		if (worker_info->stop_flag)
+			break;
+	}
+
+	rte_dma_submit(dev_id, 0);
+	while ((async_cnt > 0) && (poll_cnt++ < POLL_MAX)) {
+		nr_cpl = rte_dma_completed(dev_id, 0,

MAX_DMA_CPL_NB, NULL, NULL);

app/test-dma-perf/benchmark.c new file +6 -0
+		async_cnt -= nr_cpl;
+	}
+
+	return 0;
+}
+

<snip>

Quoted 47 lines
app/test-dma-perf/benchmark.c new file +12 -0
+static int
+setup_memory_env(struct test_configure *cfg, struct rte_mbuf ***srcs,
+			struct rte_mbuf ***dsts)
+{
+	unsigned int buf_size = cfg->buf_size.cur;
+	unsigned int nr_sockets;
+	uint32_t nr_buf = cfg->nr_buf;
+
+	nr_sockets = rte_socket_count();
+	if (cfg->src_numa_node >= nr_sockets ||
+		cfg->dst_numa_node >= nr_sockets) {
+		printf("Error: Source or destination numa exceeds the acture

numa nodes.\n");

app/test-dma-perf/benchmark.c new file +34 -0
+		return -1;
+	}
+
+	src_pool = rte_pktmbuf_pool_create("Benchmark_DMA_SRC",
+			nr_buf,
+			0,
+			0,
+			buf_size + RTE_PKTMBUF_HEADROOM,
+			cfg->src_numa_node);
+	if (src_pool == NULL) {
+		PRINT_ERR("Error with source mempool creation.\n");
+		return -1;
+	}
+
+	dst_pool = rte_pktmbuf_pool_create("Benchmark_DMA_DST",
+			nr_buf,
+			0,
+			0,
+			buf_size + RTE_PKTMBUF_HEADROOM,
+			cfg->dst_numa_node);
+	if (dst_pool == NULL) {
+		PRINT_ERR("Error with destination mempool creation.\n");
+		return -1;
+	}
+
+	*srcs = rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *), 0);
+	if (*srcs == NULL) {
+		printf("Error: srcs malloc failed.\n");
+		return -1;
+	}
+
+	*dsts = rte_malloc(NULL, nr_buf * sizeof(struct rte_mbuf *), 0);
+	if (*dsts == NULL) {
+		printf("Error: dsts malloc failed.\n");
Quoted 9 lines

app/test-dma-perf/benchmark.c line 353

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+		return -1;
+	}
+
+	if (rte_mempool_get_bulk(src_pool, (void **)*srcs, nr_buf) != 0) {

[Anoob] Might be better to use 'rte_pktmbuf_alloc_bulk' since we use ' rte_mbuf_data_iova' in the datapath and it is desirable to initialize it properly as an mbuf. Same comment below as well.

[Cheng] sure, I'll fix it in the next version.

Quoted 17 lines

<snip>

Review

Quoted 11 lines
app/test-dma-perf/benchmark.c new file +7 -0
+
+	for (i = 0; i < nb_workers; i++) {
+		calc_result(buf_size, nr_buf, nb_workers, test_secs,
+			lcores_p[i].v_ptr->worker_info.test_cpl,
+			&memory, &avg_cycles, &bandwidth, &mops);
+		output_result(cfg->scenario_id, lcores_p[i].v_ptr->lcore_id,
+					lcores_p[i].v_ptr->dma_name,

avg_cycles, buf_size,

(quoted diff) +1 -0
+					nr_buf / nb_workers, memory,

bandwidth, mops, is_dma);

(quoted diff) +1 -0
+	}

[Anoob] Can you also print total_bandwidth & total_mops? It can be a simple aggregation in the above loop. Would help when we are dealing with larger number of queues but single hardware block.

[Cheng] sure, good point. I'll add it in the next version, thanks.

Quoted 9 lines

app/test-dma-perf/benchmark.c line 466

Quoted 5 lines
app/test-dma-perf/benchmark.c new file +4 -0
+
+out:
+	/* free mbufs used in the test */
+	if (srcs)

[Anoob] DPDK coding guidelines recommend a usage like below,

	(if (srcs != NULL)

[Cheng] sure, thanks. I'll fix it in the next version.

Quoted 11 lines
Quoted 11 lines
app/test-dma-perf/benchmark.c new file +11 -0
+		rte_pktmbuf_free_bulk(srcs, nr_buf);
+	if (dsts)
+		rte_pktmbuf_free_bulk(dsts, nr_buf);
+
+	/* free the points for the mbufs */
+	rte_free(srcs);
+	srcs = NULL;
+	rte_free(dsts);
+	dsts = NULL;
+
+	if (src_pool) {
Quoted 8 lines

app/test-dma-perf/benchmark.c line 483

Quoted 6 lines
app/test-dma-perf/benchmark.c new file +6 -0
+		rte_mempool_free(src_pool);
+		src_pool = NULL;
+	}
+	if (dst_pool) {
+		rte_mempool_free(dst_pool);
+		src_pool = NULL;

[Anoob] Should be dst_pool, right?

[Cheng] yes, sorry for the miss. I'll fix it in the next version.

Quoted 13 lines

<snip>

Quoted 10 lines
app/test-dma-perf/main.h new renamed from app/test-dma-perf/main.h +0 -0

file mode 100644 index 0000000000..215ac42673

app/test-dma-perf/main.h new file +5 -0
@@ -0,0 +1,69 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 Intel Corporation  */
+
+#ifndef _MAIN_H_
+#define _MAIN_H_
Quoted 8 lines

app/test-dma-perf/main.h line 12

Quoted 6 lines
app/test-dma-perf/main.h new file +6 -0
+
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_dev.h>
+#include <rte_dmadev.h>

[Anoob] Is the above include (rte_dmadev.h) required?

[Cheng] you are right. It's not required. I'll remove it in the next version.

Quoted 8 lines

app/test-dma-perf/main.h line 16

Quoted 5 lines
app/test-dma-perf/main.h new file +4 -0
+
+#ifndef __maybe_unused
+#define __maybe_unused	__rte_unused
+#endif

[Anoob] Can you try to avoid this and use rte_unused or RTE_SET_USED in the cache_flush_buf() function?

[Cheng] sure, I'll fix it in the next version.

#37 lore raw

Hi Cheng,

Please see inline.

Thanks,
Anoob

Quoted 135 lines

-----Original Message-----
From: Jiang, Cheng1 <[email protected]>
Sent: Saturday, June 24, 2023 5:23 PM
To: Anoob Joseph <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; [email protected]; Laatz, Kevin <[email protected]>; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>
Subject: RE: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 7 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 23, 2023 2:53 PM
To: Jiang, Cheng1 <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>;

[email protected];

Quoted 4 lines

Laatz, Kevin <[email protected]>; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; Wang,

YuanX

Quoted 1 line

<[email protected]>; He, Xingguang <[email protected]>;

Ling,

Quoted 4 lines

WeiX <[email protected]>
Subject: RE: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

Quoted 12 lines

Hi Cheng,

Thanks for the new version. Please see inline.

Thanks,
Anoob

Quoted 5 lines

-----Original Message-----
From: Cheng Jiang <[email protected]>
Sent: Tuesday, June 20, 2023 12:24 PM
To: [email protected]; [email protected]; [email protected]; [email protected]; Amit Prakash

Shukla

Quoted 6 lines
Quoted 6 lines

<[email protected]>; Anoob Joseph <[email protected]>; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 74 lines
Quoted 9 lines

[email protected]; Cheng Jiang <[email protected]>
Subject: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

External Email

--------------------------------------------------------------------
-- There are many high-performance DMA devices supported in DPDK now,

and

Quoted 10 lines

these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre- set config file. Memory Copy performance test

are supported for now.

Quoted 52 lines

---
v8:
fixed string copy issue in parse_lcore();
improved some data display format;
added doc in doc/guides/tools;
updated release notes;

v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 498 +++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 594 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  69 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1350 insertions(+)  create mode 100644

app/test-dma-perf/benchmark.c create mode 100644 app/test-dma-perf/config.ini create mode 100644 app/test-dma- perf/main.c create mode 100644 app/test-dma-perf/main.h create mode
100644 app/test-dma-perf/meson.build create mode 100644 doc/guides/tools/dmaperf.rst

<snip>

Quoted 49 lines
Quoted 40 lines
Quoted 40 lines
(quoted diff) +1 -0
+			fprintf(stderr, "Error: Fail to find DMA %s.\n",

dma_name);

app/test-dma-perf/benchmark.c new file +11 -0
+			goto end;
+		}
+
+		ldm->dma_ids[i] = dev_id;
+		configure_dmadev_queue(dev_id, ring_size);
+		++nb_dmadevs;
+	}
+
+end:
+	if (nb_dmadevs < nb_workers) {
+		printf("Not enough dmadevs (%u) for all workers (%u).\n",

nb_dmadevs, nb_workers);

app/test-dma-perf/benchmark.c new file +22 -0
+		return -1;
+	}
+
+	printf("Number of used dmadevs: %u.\n", nb_dmadevs);
+
+	return 0;
+}
+
+static inline void
+do_dma_submit_and_poll(uint16_t dev_id, uint64_t *async_cnt,
+			volatile struct worker_info *worker_info) {
+	int ret;
+	uint16_t nr_cpl;
+
+	ret = rte_dma_submit(dev_id, 0);
+	if (ret < 0) {
+		rte_dma_stop(dev_id);
+		rte_dma_close(dev_id);
+		rte_exit(EXIT_FAILURE, "Error with dma submit.\n");
+	}
+
+	nr_cpl = rte_dma_completed(dev_id, 0, MAX_DMA_CPL_NB, NULL,

NULL);

app/test-dma-perf/benchmark.c new file +2 -0
+	*async_cnt -= nr_cpl;
+	worker_info->total_cpl += nr_cpl;
Quoted 8 lines

app/test-dma-perf/benchmark.c line 214

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+}
+
+static inline int
+do_dma_mem_copy(void *p)

[Anoob] Just curious, why not pass struct lcore_params *para itself? Is it because the pointer is volatile? If yes, then we can take an AI to split the struct into volatile and non-volatile parts.

Quoted 5 lines

[Cheng] The reason I did it this way is because I want to launch this function on another core by spawning a new thread, and rte_eal_remote_launch() takes a void * as the parameter. That's why I passed void *p. Your suggestion to split the struct into volatile and non-volatile parts is quite reasonable. I am thinking about the best way to implement it. Thanks.

[Anoob] Instead of passing the address of index variable as void *, you can easily send lcore_params pointer, right?

Quoted 62 lines
Quoted 25 lines
Quoted 25 lines
app/test-dma-perf/benchmark.c new file +23 -0
+{
+	const uint16_t *para_idx = (uint16_t *)p;
+	volatile struct lcore_params *para = lcores_p[*para_idx].v_ptr;
+	volatile struct worker_info *worker_info = &(para->worker_info);
+	const uint16_t dev_id = para->dev_id;
+	const uint32_t nr_buf = para->nr_buf;
+	const uint16_t kick_batch = para->kick_batch;
+	const uint32_t buf_size = para->buf_size;
+	struct rte_mbuf **srcs = para->srcs;
+	struct rte_mbuf **dsts = para->dsts;
+	uint16_t nr_cpl;
+	uint64_t async_cnt = 0;
+	uint32_t i;
+	uint32_t poll_cnt = 0;
+	int ret;
+
+	worker_info->stop_flag = false;
+	worker_info->ready_flag = true;
+
+	while (!worker_info->start_flag)
+		;
+
+	while (1) {
Quoted 8 lines

app/test-dma-perf/benchmark.c line 241

Quoted 5 lines
app/test-dma-perf/benchmark.c new file +3 -0
+		for (i = 0; i < nr_buf; i++) {
+dma_copy:
+			ret = rte_dma_copy(dev_id, 0,

rte_pktmbuf_iova(srcs[i]),

app/test-dma-perf/benchmark.c new file +1 -0
+				rte_pktmbuf_iova(dsts[i]), buf_size, 0);

[Anoob] Do we need to use ' rte_mbuf_data_iova' here instead of 'rte_pktmbuf_iova'?

[Cheng] yes rte_mbuf_data_iova is more appropriate, I'll fix it in the next version. Thanks.

Quoted 16 lines

app/test-dma-perf/benchmark.c line 248

Quoted 9 lines
app/test-dma-perf/benchmark.c new file +3 -0
+			if (unlikely(ret < 0)) {
+				if (ret == -ENOSPC) {
+					do_dma_submit_and_poll(dev_id,

&async_cnt, worker_info);

app/test-dma-perf/benchmark.c new file +4 -0
+					goto dma_copy;
+				} else {
+					/* Error exit */
+					rte_dma_stop(dev_id);

[Anoob] Missing rte_dma_close() here. Also, may be introduce a static void function so that rte_exit call etc won't be part of the fastpath loop.

May be something like below and you can call it from here and "do_dma_submit_and_poll".

Quoted 8 lines

static void
error_exit(int dev_id)

{
	/* Error exit */
	rte_dma_stop(dev_id);
	rte_dma_close(dev_id);
	rte_exit(EXIT_FAILURE, "DMA enqueue failed\n"); }
Quoted 2 lines

[Cheng] I'm not so sure here. rte_dma_close() is called in the rte_exit(). Do we still call it explicitly before rte_exit()?

[Anoob] In ' do_dma_submit_and_poll', there is rte_dma_close() before rte_exit(). I'm fine either way as long is it is consistent. Said that, I think it is better to call close() from app, rather than relying on rte_exit.

<snip>

#38 lore raw

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 218 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Monday, June 26, 2023 1:42 PM
To: Jiang, Cheng1 <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>; [email protected]; Laatz, Kevin <[email protected]>; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>
Subject: RE: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

Hi Cheng,

Please see inline.

Thanks,
Anoob

Quoted 7 lines

-----Original Message-----
From: Jiang, Cheng1 <[email protected]>
Sent: Saturday, June 24, 2023 5:23 PM
To: Anoob Joseph <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; Amit Prakash Shukla <[email protected]>;

[email protected];

Quoted 4 lines

Laatz, Kevin <[email protected]>; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; Wang,

YuanX

Quoted 5 lines

<[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>
Subject: RE: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

Quoted 13 lines

Hi Anoob,

Replies are inline.

Thanks,
Cheng

Quoted 6 lines

-----Original Message-----
From: Anoob Joseph <[email protected]>
Sent: Friday, June 23, 2023 2:53 PM
To: Jiang, Cheng1 <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>;

Amit

Quoted 108 lines
Quoted 1 line

Prakash Shukla <[email protected]>;

[email protected];

Quoted 4 lines

Laatz, Kevin <[email protected]>; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; Wang,

YuanX

Quoted 1 line

<[email protected]>; He, Xingguang <[email protected]>;

Ling,

Quoted 4 lines

WeiX <[email protected]>
Subject: RE: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

Quoted 12 lines

Hi Cheng,

Thanks for the new version. Please see inline.

Thanks,
Anoob

Quoted 5 lines

-----Original Message-----
From: Cheng Jiang <[email protected]>
Sent: Tuesday, June 20, 2023 12:24 PM
To: [email protected]; [email protected]; [email protected]; [email protected]; Amit Prakash

Shukla

Quoted 6 lines
Quoted 6 lines

<[email protected]>; Anoob Joseph <[email protected]>; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected];

[email protected];

Quoted 75 lines
Quoted 10 lines

[email protected]; Cheng Jiang <[email protected]>
Subject: [EXT] [PATCH v8] app/dma-perf: introduce dma-perf application

External Email

------------------------------------------------------------------
--
-- There are many high-performance DMA devices supported in DPDK now,

and

Quoted 10 lines

these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre- set config file. Memory Copy performance test

are supported for now.

Quoted 52 lines

---
v8:
fixed string copy issue in parse_lcore();
improved some data display format;
added doc in doc/guides/tools;
updated release notes;

v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 498 +++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 594 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  69 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1350 insertions(+)  create mode 100644

app/test-dma-perf/benchmark.c create mode 100644 app/test-dma-perf/config.ini create mode 100644 app/test-dma- perf/main.c create mode 100644 app/test-dma-perf/main.h create mode
100644 app/test-dma-perf/meson.build create mode 100644 doc/guides/tools/dmaperf.rst

<snip>

Quoted 48 lines
Quoted 39 lines
Quoted 39 lines
(quoted diff) +1 -0
+			fprintf(stderr, "Error: Fail to find DMA %s.\n",

dma_name);

app/test-dma-perf/benchmark.c new file +11 -0
+			goto end;
+		}
+
+		ldm->dma_ids[i] = dev_id;
+		configure_dmadev_queue(dev_id, ring_size);
+		++nb_dmadevs;
+	}
+
+end:
+	if (nb_dmadevs < nb_workers) {
+		printf("Not enough dmadevs (%u) for all workers (%u).\n",

nb_dmadevs, nb_workers);

app/test-dma-perf/benchmark.c new file +22 -0
+		return -1;
+	}
+
+	printf("Number of used dmadevs: %u.\n", nb_dmadevs);
+
+	return 0;
+}
+
+static inline void
+do_dma_submit_and_poll(uint16_t dev_id, uint64_t *async_cnt,
+			volatile struct worker_info *worker_info) {
+	int ret;
+	uint16_t nr_cpl;
+
+	ret = rte_dma_submit(dev_id, 0);
+	if (ret < 0) {
+		rte_dma_stop(dev_id);
+		rte_dma_close(dev_id);
+		rte_exit(EXIT_FAILURE, "Error with dma submit.\n");
+	}
+
+	nr_cpl = rte_dma_completed(dev_id, 0, MAX_DMA_CPL_NB, NULL,

NULL);

app/test-dma-perf/benchmark.c new file +1 -0
+	*async_cnt -= nr_cpl;
Quoted 8 lines

app/test-dma-perf/benchmark.c line 214

Quoted 4 lines
app/test-dma-perf/benchmark.c new file +4 -0
+	worker_info->total_cpl += nr_cpl; }
+
+static inline int
+do_dma_mem_copy(void *p)

[Anoob] Just curious, why not pass struct lcore_params *para itself? Is it because the pointer is volatile? If yes, then we can take an AI to split the struct into volatile and non-volatile parts.

Quoted 5 lines

[Cheng] The reason I did it this way is because I want to launch this function on another core by spawning a new thread, and rte_eal_remote_launch() takes a void * as the parameter. That's why I passed void *p. Your suggestion to split the struct into volatile and non-volatile parts is quite reasonable. I am thinking about the best way to

Quoted 5 lines

implement it. Thanks.

[Anoob] Instead of passing the address of index variable as void *, you can easily send lcore_params pointer, right?

[Cheng] Yes, you are right. I can pass the lcore_params pointer. And I'll fix it in the next version. The new lcore_params will be non-volatile with volatile worker_info in it. This is more reasonable.

Quoted 63 lines
Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
app/test-dma-perf/benchmark.c new file +23 -0
+{
+	const uint16_t *para_idx = (uint16_t *)p;
+	volatile struct lcore_params *para = lcores_p[*para_idx].v_ptr;
+	volatile struct worker_info *worker_info = &(para->worker_info);
+	const uint16_t dev_id = para->dev_id;
+	const uint32_t nr_buf = para->nr_buf;
+	const uint16_t kick_batch = para->kick_batch;
+	const uint32_t buf_size = para->buf_size;
+	struct rte_mbuf **srcs = para->srcs;
+	struct rte_mbuf **dsts = para->dsts;
+	uint16_t nr_cpl;
+	uint64_t async_cnt = 0;
+	uint32_t i;
+	uint32_t poll_cnt = 0;
+	int ret;
+
+	worker_info->stop_flag = false;
+	worker_info->ready_flag = true;
+
+	while (!worker_info->start_flag)
+		;
+
+	while (1) {
Quoted 8 lines

app/test-dma-perf/benchmark.c line 241

Quoted 5 lines
app/test-dma-perf/benchmark.c new file +3 -0
+		for (i = 0; i < nr_buf; i++) {
+dma_copy:
+			ret = rte_dma_copy(dev_id, 0,

rte_pktmbuf_iova(srcs[i]),

app/test-dma-perf/benchmark.c new file +1 -0
+				rte_pktmbuf_iova(dsts[i]), buf_size, 0);

[Anoob] Do we need to use ' rte_mbuf_data_iova' here instead of 'rte_pktmbuf_iova'?

Quoted 15 lines

[Cheng] yes rte_mbuf_data_iova is more appropriate, I'll fix it in the next version. Thanks.

Quoted 12 lines

app/test-dma-perf/benchmark.c line 248

Quoted 9 lines
app/test-dma-perf/benchmark.c new file +3 -0
+			if (unlikely(ret < 0)) {
+				if (ret == -ENOSPC) {
+					do_dma_submit_and_poll(dev_id,

&async_cnt, worker_info);

app/test-dma-perf/benchmark.c new file +4 -0
+					goto dma_copy;
+				} else {
+					/* Error exit */
+					rte_dma_stop(dev_id);

[Anoob] Missing rte_dma_close() here. Also, may be introduce a static void function so that rte_exit call etc won't be part of the fastpath

loop.

Quoted 13 lines
Quoted 12 lines

May be something like below and you can call it from here and "do_dma_submit_and_poll".

static void
error_exit(int dev_id)

{
	/* Error exit */
	rte_dma_stop(dev_id);
	rte_dma_close(dev_id);
	rte_exit(EXIT_FAILURE, "DMA enqueue failed\n"); }
Quoted 7 lines
Quoted 2 lines

[Cheng] I'm not so sure here. rte_dma_close() is called in the rte_exit(). Do we still call it explicitly before rte_exit()?

[Anoob] In ' do_dma_submit_and_poll', there is rte_dma_close() before rte_exit(). I'm fine either way as long is it is consistent. Said that, I think it is better to call close() from app, rather than relying on rte_exit.

[Cheng] sure, it makes sense to me that app should call rte_dma_close(), I'll fix it in the next version. Thanks.

Quoted 1 line

<snip>

#39 lore raw

PATCH v9 1/1 app/dma-perf: introduce dma-perf application

In reply to Cheng Jiang

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are supported for now.

---
v9:
improved error handling;
improved lcore_params structure;
improved mbuf api calling;
improved exit process;
fixed some typos;
added scenario summary data display;
removed unnecessary include;
v8:
fixed string copy issue in parse_lcore(); improved some data display format;
added doc in doc/guides/tools;
updated release notes;
v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 508 ++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 616 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  64 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1377 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c
 create mode 100644 app/test-dma-perf/config.ini
 create mode 100644 app/test-dma-perf/main.c
 create mode 100644 app/test-dma-perf/main.h
 create mode 100644 app/test-dma-perf/meson.build
 create mode 100644 doc/guides/tools/dmaperf.rst
app/meson.build +1 -0
index 74d2420f67..4fc1a83eba 100644
@@ -19,6 +19,7 @@ apps = [
         'test-cmdline',
         'test-compress-perf',
         'test-crypto-perf',
+        'test-dma-perf',
         'test-eventdev',
         'test-fib',
         'test-flow-perf',
app/test-dma-perf/benchmark.c new file +508 -0

508 lines not on this page · raw message

app/test-dma-perf/config.ini new file +61 -0

61 lines not on this page · raw message

app/test-dma-perf/main.c new file +616 -0

616 lines not on this page · raw message

app/test-dma-perf/main.h new file +64 -0

64 lines not on this page · raw message

app/test-dma-perf/meson.build new file +17 -0
new file mode 100644
index 0000000000..bd6c264002
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2023 Intel Corporation
+
+# meson file, for building this app as part of a main DPDK build.
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+deps += ['dmadev', 'mbuf', 'cfgfile']
+
+sources = files(
+        'main.c',
+        'benchmark.c',
+)
doc/guides/rel_notes/release_23_07.rst +6 -0
index 027ae7bd2d..2355b558bc 100644
@@ -170,6 +170,12 @@ New Features
 
   See :doc:`../prog_guide/pdcp_lib` for more information.
 
+* **Added DMA device performance test application.**
+
+  Added an new application to test the performance of DMA device and CPU.
+
+  See the :doc:`../tools/dmaperf` for more details.
+
 
 Removed Items
 -------------
doc/guides/tools/dmaperf.rst new file +103 -0

103 lines not on this page · raw message

doc/guides/tools/index.rst +1 -0
index 6f84fc31ff..857572da96 100644
@@ -23,3 +23,4 @@ DPDK Tools User Guides
     testregex
     testmldev
     dts
+    dmaperf

--
2.40.1

#40 lore raw

PATCH v10 1/1 app/dma-perf: introduce dma-perf application

In reply to Cheng Jiang

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are supported for now.

---
v10:
rebased code from 23.07-rc2;
v9:
improved error handling;
improved lcore_params structure;
improved mbuf api calling;
improved exit process;
fixed some typos;
added scenario summary data display;
removed unnecessary include;
v8:
fixed string copy issue in parse_lcore(); improved some data display format;
added doc in doc/guides/tools;
updated release notes;
v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 508 ++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 616 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  64 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1377 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c
 create mode 100644 app/test-dma-perf/config.ini
 create mode 100644 app/test-dma-perf/main.c
 create mode 100644 app/test-dma-perf/main.h
 create mode 100644 app/test-dma-perf/meson.build
 create mode 100644 doc/guides/tools/dmaperf.rst
app/meson.build +1 -0
index 74d2420f67..4fc1a83eba 100644
@@ -19,6 +19,7 @@ apps = [
         'test-cmdline',
         'test-compress-perf',
         'test-crypto-perf',
+        'test-dma-perf',
         'test-eventdev',
         'test-fib',
         'test-flow-perf',
app/test-dma-perf/benchmark.c new file +508 -0

508 lines not on this page · raw message

app/test-dma-perf/config.ini new file +61 -0

61 lines not on this page · raw message

app/test-dma-perf/main.c new file +616 -0

616 lines not on this page · raw message

app/test-dma-perf/main.h new file +64 -0

64 lines not on this page · raw message

app/test-dma-perf/meson.build new file +17 -0
new file mode 100644
index 0000000000..bd6c264002
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2023 Intel Corporation
+
+# meson file, for building this app as part of a main DPDK build.
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+deps += ['dmadev', 'mbuf', 'cfgfile']
+
+sources = files(
+        'main.c',
+        'benchmark.c',
+)
doc/guides/rel_notes/release_23_07.rst +6 -0
index 4459144140..796cc5517d 100644
@@ -200,6 +200,12 @@ New Features
 
   Enhanced the GRO library to support TCP packets over IPv6 network.
 
+* **Added DMA device performance test application.**
+
+  Added an new application to test the performance of DMA device and CPU.
+
+  See the :doc:`../tools/dmaperf` for more details.
+
 
 Removed Items
 -------------
doc/guides/tools/dmaperf.rst new file +103 -0

103 lines not on this page · raw message

doc/guides/tools/index.rst +1 -0
index 6f84fc31ff..857572da96 100644
@@ -23,3 +23,4 @@ DPDK Tools User Guides
     testregex
     testmldev
     dts
+    dmaperf

--
2.40.1

Acked-by Tested-by on v10 1/1
#41 lore raw
Quoted 65 lines

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre- set config file. Memory Copy performance test are supported for now.

---
v10:
rebased code from 23.07-rc2;
v9:
improved error handling;
improved lcore_params structure;
improved mbuf api calling;
improved exit process;
fixed some typos;
added scenario summary data display;
removed unnecessary include;
v8:
fixed string copy issue in parse_lcore(); improved some data display format;
added doc in doc/guides/tools;
updated release notes;
v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 508 ++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 616 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  64 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1377 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c  create mode 100644
Quoted 5 lines

app/test-dma-perf/config.ini create mode 100644 app/test-dma- perf/main.c create mode 100644 app/test-dma-perf/main.h create mode 100644 app/test-dma-perf/meson.build create mode 100644 doc/guides/tools/dmaperf.rst

Thanks Cheng for addressing all the comments.

Tested-by on v10 1/1
#42 lore raw
In reply to Cheng Jiang
Quoted 79 lines

-----Original Message-----
From: Jiang, Cheng1 <[email protected]>
Sent: Wednesday, June 28, 2023 9:21 AM
To: [email protected]; Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; [email protected]; [email protected]; [email protected]; Laatz, Kevin <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>; Jiang, Cheng1 <[email protected]>
Subject: [PATCH v10] app/dma-perf: introduce dma-perf application

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre- set config file. Memory Copy performance test are supported for now.

---
v10:
rebased code from 23.07-rc2;
v9:
improved error handling;
improved lcore_params structure;
improved mbuf api calling;
improved exit process;
fixed some typos;
added scenario summary data display;
removed unnecessary include;
v8:
fixed string copy issue in parse_lcore(); improved some data display format;
added doc in doc/guides/tools;
updated release notes;
v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 508 ++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 616 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  64 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 9 files changed, 1377 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c  create mode 100644
Quoted 5 lines

app/test-dma-perf/config.ini create mode 100644 app/test-dma- perf/main.c create mode 100644 app/test-dma-perf/main.h create mode 100644 app/test-dma-perf/meson.build create mode 100644 doc/guides/tools/dmaperf.rst

#43 lore raw
In reply to Cheng Jiang

28/06/2023 03:20, Cheng Jiang:

Quoted 11 lines

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are supported for now.

Quoted 5 lines

Who is going to be the maintainer for this new app? An entry in the file MAINTAINERS would be perfect.

#44 lore raw

Hi Thomas,

Quoted 28 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, June 29, 2023 5:09 PM
To: Hu, Jiayu <[email protected]>; Wang, YuanX <[email protected]>; Jiang, Cheng1 <[email protected]>
Cc: Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; [email protected]; [email protected]; [email protected]; Laatz, Kevin <[email protected]>; [email protected]; [email protected]; [email protected]; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>
Subject: Re: [PATCH v10] app/dma-perf: introduce dma-perf application

28/06/2023 03:20, Cheng Jiang:

Quoted 1 line

There are many high-performance DMA devices supported in DPDK now,

and

Quoted 9 lines

these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are

supported for now.

Quoted 1 line
Quoted 9 lines
Quoted 5 lines

Who is going to be the maintainer for this new app? An entry in the file MAINTAINERS would be perfect.

I can be the maintainer for this new app. Okay, I will add an entry to the MAINTAINERS file. Would you prefer I send a new version patch to add this entry, or send a separate patch?

Thanks,
Cheng

Quoted 1 line
#45 lore raw

29/06/2023 14:50, Jiang, Cheng1:

Quoted 21 lines

From: Thomas Monjalon <[email protected]>

Quoted 20 lines

28/06/2023 03:20, Cheng Jiang:

Quoted 1 line

There are many high-performance DMA devices supported in DPDK now,

and

Quoted 9 lines

these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are

supported for now.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Who is going to be the maintainer for this new app? An entry in the file MAINTAINERS would be perfect.

I can be the maintainer for this new app. Okay, I will add an entry to the MAINTAINERS file. Would you prefer I send a new version patch to add this entry, or send a separate patch?

It should be part of this patch.

#46 lore raw
Quoted 44 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, June 29, 2023 9:19 PM
To: Hu, Jiayu <[email protected]>; Wang, YuanX <[email protected]>; Jiang, Cheng1 <[email protected]>
Cc: Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; [email protected]; [email protected]; [email protected]; Laatz, Kevin <[email protected]>; [email protected]; [email protected]; [email protected]; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>
Subject: Re: [PATCH v10] app/dma-perf: introduce dma-perf application

29/06/2023 14:50, Jiang, Cheng1:

Quoted 3 lines

From: Thomas Monjalon <[email protected]>

Quoted 2 lines

28/06/2023 03:20, Cheng Jiang:

Quoted 1 line

There are many high-performance DMA devices supported in DPDK

now,

Quoted 23 lines
Quoted 22 lines

and

Quoted 10 lines

these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are

supported for now.

Quoted 6 lines

Who is going to be the maintainer for this new app? An entry in the file MAINTAINERS would be perfect.

Quoted 8 lines
Quoted 1 line

I can be the maintainer for this new app. Okay, I will add an entry to the

MAINTAINERS file.

Quoted 1 line

Would you prefer I send a new version patch to add this entry, or send a

separate patch?

It should be part of this patch.

Sure, I'll submit a new patch. Thanks.

Quoted 2 lines
#47 lore raw
In reply to Cheng Jiang

28/06/2023 03:20, Cheng Jiang:

doc/guides/tools/index.rst DPDK Tools User Guides · line 26

Quoted 7 lines
doc/guides/tools/index.rst +1 -0
@@ -23,3 +23,4 @@ DPDK Tools User Guides
     testregex
     testmldev
     dts
+    dmaperf

Would be better to have upper in the list (DTS should be last). I suggest before flow-perf as DMA is more low-level.

#48 lore raw

Hi Thomas,

Quoted 15 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, June 29, 2023 5:38 PM
To: Jiang, Cheng1 <[email protected]>
Cc: Richardson, Bruce <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>; [email protected]; [email protected]; [email protected]; Laatz, Kevin <[email protected]>; [email protected]; [email protected]; [email protected]; Hu, Jiayu <[email protected]>; Ding, Xuan <[email protected]>; Ma, WenwuX <[email protected]>; Wang, YuanX <[email protected]>; He, Xingguang <[email protected]>; Ling, WeiX <[email protected]>
Subject: Re: [PATCH v10] app/dma-perf: introduce dma-perf application

28/06/2023 03:20, Cheng Jiang:

Quoted 11 lines

doc/guides/tools/index.rst DPDK Tools User Guides · line 26

Quoted 7 lines
doc/guides/tools/index.rst +1 -0
@@ -23,3 +23,4 @@ DPDK Tools User Guides
     testregex
     testmldev
     dts
+    dmaperf

Would be better to have upper in the list (DTS should be last). I suggest before flow-perf as DMA is more low-level.

Sure, I'll submit a new patch to fix it.

Thanks,
Cheng

#49 lore raw

PATCH v11 1/1 app/dma-perf: introduce dma-perf application

In reply to Cheng Jiang

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are supported for now. This patch also updates the documentation and maintainer list for the application.

---
v11:
updated MAINTAINERS file;
fixed typo in doc/guides/tools/dmaperf.rst; changed the dmaperf order in doc/guides/tools/index.rst;
v10:
rebased code from 23.07-rc2;
v9:
improved error handling;
improved lcore_params structure;
improved mbuf api calling;
improved exit process;
fixed some typos;
added scenario summary data display;
removed unnecessary include;
v8:
fixed string copy issue in parse_lcore(); improved some data display format;
added doc in doc/guides/tools;
updated release notes;
v7:
fixed some strcpy issues;
removed cache setup in calling rte_pktmbuf_pool_create(); fixed some typos;
added some memory free and null set operations; improved result calculation;
v6:
improved code based on Anoob's comments;
fixed some code structure issues;
v5:
fixed some LONG_LINE warnings;
v4:
fixed inaccuracy of the memory footprint display;
v3:
fixed some typos;
v2:
added lcore/dmadev designation;
added error case process;
removed worker_threads parameter from config.ini; improved the logs;
improved config file;

 MAINTAINERS                            |   5 +
 app/meson.build                        |   1 +
 app/test-dma-perf/benchmark.c          | 508 ++++++++++++++++++++
 app/test-dma-perf/config.ini           |  61 +++
 app/test-dma-perf/main.c               | 616 +++++++++++++++++++++++++
 app/test-dma-perf/main.h               |  64 +++
 app/test-dma-perf/meson.build          |  17 +
 doc/guides/rel_notes/release_23_07.rst |   6 +
 doc/guides/tools/dmaperf.rst           | 103 +++++
 doc/guides/tools/index.rst             |   1 +
 10 files changed, 1382 insertions(+)
 create mode 100644 app/test-dma-perf/benchmark.c
 create mode 100644 app/test-dma-perf/config.ini
 create mode 100644 app/test-dma-perf/main.c
 create mode 100644 app/test-dma-perf/main.h
 create mode 100644 app/test-dma-perf/meson.build
 create mode 100644 doc/guides/tools/dmaperf.rst
MAINTAINERS +5 -0
index 545ed6dd7d..b99d318608 100644
@@ -1767,6 +1767,11 @@ T: git://dpdk.org/next/dpdk-next-net
 F: app/test-pmd/
 F: doc/guides/testpmd_app_ug/
 
+DMA device performance tool
+M: Cheng Jiang <[email protected]>
+F: app/test-dma-perf/
+F: doc/guides/tools/dmaperf.rst
+
 Flow performance tool
 M: Wisam Jaddo <[email protected]>
 F: app/test-flow-perf/
app/meson.build +1 -0
index 74d2420f67..4fc1a83eba 100644
@@ -19,6 +19,7 @@ apps = [
         'test-cmdline',
         'test-compress-perf',
         'test-crypto-perf',
+        'test-dma-perf',
         'test-eventdev',
         'test-fib',
         'test-flow-perf',
app/test-dma-perf/benchmark.c new file +508 -0

508 lines not on this page · raw message

app/test-dma-perf/config.ini new file +61 -0

61 lines not on this page · raw message

app/test-dma-perf/main.c new file +616 -0

616 lines not on this page · raw message

app/test-dma-perf/main.h new file +64 -0

64 lines not on this page · raw message

app/test-dma-perf/meson.build new file +17 -0
new file mode 100644
index 0000000000..bd6c264002
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2019-2023 Intel Corporation
+
+# meson file, for building this app as part of a main DPDK build.
+
+if is_windows
+    build = false
+    reason = 'not supported on Windows'
+    subdir_done()
+endif
+
+deps += ['dmadev', 'mbuf', 'cfgfile']
+
+sources = files(
+        'main.c',
+        'benchmark.c',
+)
doc/guides/rel_notes/release_23_07.rst +6 -0
index 4459144140..796cc5517d 100644
@@ -200,6 +200,12 @@ New Features
 
   Enhanced the GRO library to support TCP packets over IPv6 network.
 
+* **Added DMA device performance test application.**
+
+  Added an new application to test the performance of DMA device and CPU.
+
+  See the :doc:`../tools/dmaperf` for more details.
+
 
 Removed Items
 -------------
doc/guides/tools/dmaperf.rst new file +103 -0

103 lines not on this page · raw message

doc/guides/tools/index.rst +1 -0
index 6f84fc31ff..f2afb1fcc5 100644
@@ -14,6 +14,7 @@ DPDK Tools User Guides
     pmdinfo
     dumpcap
     pdump
+    dmaperf
     flow-perf
     securityperf
     testbbdev

--
2.40.1

Acked-by on v11 1/1
#50 lore raw
Quoted 21 lines · Cheng Jiang

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are supported for now. This patch also updates the documentation and maintainer list for the application.

...

Quoted 3 lines

.

#51 lore raw
Quoted 22 lines
Quoted 13 lines

There are many high-performance DMA devices supported in DPDK now, and these DMA devices can also be integrated into other modules of DPDK as accelerators, such as Vhost. Before integrating DMA into applications, developers need to know the performance of these DMA devices in various scenarios and the performance of CPUs in the same scenario, such as different buffer lengths. Only in this way can we know the target performance of the application accelerated by using them. This patch introduces a high-performance testing tool, which supports comparing the performance of CPU and DMA in different scenarios automatically with a pre-set config file. Memory Copy performance test are supported for now. This patch also updates the documentation and maintainer list for the application.

Quoted 8 lines
Quoted 1 line

Applied with few improvements in the documentation, thanks.