Skip to contentSkip to conversation

test: allow taking test names from commandline

PATCH In review Bruce Richardson 3 participants 8 messages Jan 27, 2021 – Apr 14, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 test: allow taking test names from commandline

While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's often easier to specify the test name as a commandline parameter to a test binary. This also allows the test runs to be saved as part of the shell cmdline history.

This patch adds support for checking all parameters after the EAL ones, and running all valid autotests requested - either from DPDK_TEST or on the commandline. This also allows multiple tests to be run in a single automated session, which is useful for working with components which have multiple test suites.

---
 app/test/commands.c | 11 +++++++++++
 app/test/test.c     | 40 +++++++++++++++++++++++++++++++++-------
 app/test/test.h     |  1 +
 3 files changed, 45 insertions(+), 7 deletions(-)
app/test/commands.c +11 -0
index d48dd513d7..76f6ee5d23 100644
@@ -381,3 +381,14 @@ int commands_init(void)
 	cmd_autotest_autotest.string_data.str = commands;
 	return 0;
 }
+
+int command_valid(const char *cmd)
+{
+	struct test_command *t;
+
+	TAILQ_FOREACH(t, &commands_list, next) {
+		if (strcmp(t->command, cmd) == 0)
+			return 1;
+	}
+	return 0;
+}
app/test/test.c +33 -7
index 624dd48042..b4ef2bf22b 100644
@@ -93,6 +93,9 @@ main(int argc, char **argv)
 {
 #ifdef RTE_LIB_CMDLINE
 	struct cmdline *cl;
+	char *tests[argc]; /* store an array of tests to run */
+	int test_count = 0;
+	int i;
 #endif
 	char *extra_args;
 	int ret;
@@ -147,6 +150,7 @@ main(int argc, char **argv)
 	}
 
 	argv += ret;
+	argc -= ret;
 
 	prgname = argv[0];
 
@@ -165,7 +169,25 @@ main(int argc, char **argv)
 
 #ifdef RTE_LIB_CMDLINE
 	char *dpdk_test = getenv("DPDK_TEST");
-	if (dpdk_test && strlen(dpdk_test)) {
+
+	if (dpdk_test && strlen(dpdk_test) == 0)
+		dpdk_test = NULL;
+
+	if (dpdk_test && !command_valid(dpdk_test)) {
+		RTE_LOG(WARNING, APP, "Invalid DPDK_TEST value '%s'\n", dpdk_test);
+		dpdk_test = NULL;
+	}
+
+	if (dpdk_test)
+		tests[test_count++] = dpdk_test;
+	for (i = 1; i < argc; i++) {
+		if (!command_valid(argv[i]))
+			RTE_LOG(WARNING, APP, "Invalid test requested: '%s'\n", argv[i]);
+		else
+			tests[test_count++] = argv[i];
+	}
+
+	if (test_count > 0) {
 		char buf[1024];
 
 		cl = cmdline_new(main_ctx, "RTE>>", 0, 1);
@@ -174,13 +196,17 @@ main(int argc, char **argv)
 			goto out;
 		}
 
-		snprintf(buf, sizeof(buf), "%s\n", dpdk_test);
-		if (cmdline_in(cl, buf, strlen(buf)) < 0) {
-			printf("error on cmdline input\n");
+		for (i = 0; i < test_count; i++) {
+			snprintf(buf, sizeof(buf), "%s\n", tests[i]);
+			if (cmdline_in(cl, buf, strlen(buf)) < 0) {
+				printf("error on cmdline input\n");
 
-			ret = -1;
-		} else {
-			ret = last_test_result;
+				ret = -1;
+			} else
+				ret = last_test_result;
+
+			if (ret != 0)
+				break;
 		}
 		cmdline_free(cl);
 		goto out;
app/test/test.h +1 -0
index b07f6c1ef0..cd047eb26c 100644
@@ -152,6 +152,7 @@ extern int last_test_result;
 extern const char *prgname;
 
 int commands_init(void);
+int command_valid(const char *cmd);
 
 int test_mp_secondary(void);
 int test_timer_secondary(void);

--
2.27.0

Acked-by on v1 1/1
#1 lore raw
Quoted 5 lines · Bruce Richardson

While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's often easier to specify the test name as a commandline parameter to a test binary. This also allows the test runs to be saved as part of the shell cmdline history.

Quoted 8 lines

This patch adds support for checking all parameters after the EAL ones, and running all valid autotests requested - either from DPDK_TEST or on the commandline. This also allows multiple tests to be run in a single automated session, which is useful for working with components which have multiple test suites.

---

LGTM, and I think it can be useful for running specific tests when developers make changes.

#2 lore raw
Quoted 8 lines · Aaron Conole
Quoted 5 lines · Bruce Richardson

While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's often easier to specify the test name as a commandline parameter to a test binary. This also allows the test runs to be saved as part of the shell cmdline history.

Quoted 8 lines
Quoted 7 lines

This patch adds support for checking all parameters after the EAL ones, and running all valid autotests requested - either from DPDK_TEST or on the commandline. This also allows multiple tests to be run in a single automated session, which is useful for working with components which have multiple test suites.

Applied with updated commitlog, thanks.

Signature · 1 line
David Marchand
#3 lore raw
In reply to Bruce Richardson
Quoted 5 lines · Bruce Richardson

While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's often easier to specify the test name as a commandline parameter to a test binary. This also allows the test runs to be saved as part of the shell cmdline history.

I don't get the argument about history:

$ history |grep DPDK_TEST
10615  2021-03-24 10:42:11 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10636  2021-03-24 10:51:09 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10653  2021-03-24 11:17:01 ninja-build -C build -j4 &&
DPDK_TEST=kvargs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10794  2021-03-25 18:37:48 history |grep DPDK_TEST
Quoted 6 lines

This patch adds support for checking all parameters after the EAL ones, and running all valid autotests requested - either from DPDK_TEST or on the commandline. This also allows multiple tests to be run in a single automated session, which is useful for working with components which have multiple test suites.

The same could be achieved splitting DPDK_TEST content with spaces, since test names don't contain one.

Signature · 1 line
David Marchand
#4 lore raw
Quoted 10 lines · David Marchand
Quoted 5 lines · Bruce Richardson

While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's often easier to specify the test name as a commandline parameter to a test binary. This also allows the test runs to be saved as part of the shell cmdline history.

I don't get the argument about history:

Quoted 12 lines
$ history |grep DPDK_TEST
10615  2021-03-24 10:42:11 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10636  2021-03-24 10:51:09 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10653  2021-03-24 11:17:01 ninja-build -C build -j4 &&
DPDK_TEST=kvargs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10794  2021-03-25 18:37:48 history |grep DPDK_TEST

Sure, if you always specify the test name explicitly for each command, rather than running the one test multiple times having set it separately in the environment.
Overall, though I take the point that from a history saving point of view it's a minor saving.

Quoted 11 lines
Quoted 7 lines

This patch adds support for checking all parameters after the EAL ones, and running all valid autotests requested - either from DPDK_TEST or on the commandline. This also allows multiple tests to be run in a single automated session, which is useful for working with components which have multiple test suites.

The same could be achieved splitting DPDK_TEST content with spaces, since test names don't contain one.

Yep, that's a useful enhancement too, but I still thing it's better to just have the list of tests appended to the test binary command rather than have to be worrying about properly quoting a specific environment variable at the start of each command.

/Bruce

#5 lore raw
Quoted 38 lines · Bruce Richardson
Quoted 10 lines · David Marchand
Quoted 5 lines · Bruce Richardson

While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's often easier to specify the test name as a commandline parameter to a test binary. This also allows the test runs to be saved as part of the shell cmdline history.

I don't get the argument about history:

Quoted 12 lines
$ history |grep DPDK_TEST
10615  2021-03-24 10:42:11 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10636  2021-03-24 10:51:09 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10653  2021-03-24 11:17:01 ninja-build -C build -j4 &&
DPDK_TEST=kvargs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10794  2021-03-25 18:37:48 history |grep DPDK_TEST

Sure, if you always specify the test name explicitly for each command, rather than running the one test multiple times having set it separately in the environment.
Overall, though I take the point that from a history saving point of view it's a minor saving.

Quoted 8 lines
Quoted 7 lines

This patch adds support for checking all parameters after the EAL ones, and running all valid autotests requested - either from DPDK_TEST or on the commandline. This also allows multiple tests to be run in a single automated session, which is useful for working with components which have multiple test suites.

Quoted 9 lines
Quoted 3 lines

The same could be achieved splitting DPDK_TEST content with spaces, since test names don't contain one.

Yep, that's a useful enhancement too, but I still thing it's better to just have the list of tests appended to the test binary command rather than have to be worrying about properly quoting a specific environment variable at the start of each command.

Ping on this.

Other instances where using cmdline is preferred over environment

* Calling tests using sudo.
* Calling tests as an execute action when doing a git rebase

Yes, again in both cases, other workarounds are generally available (e.g. sudo -E, and exporting to environemtn before rebase), but also generally less convenient.

/Bruce

#6 lore raw
Quoted 44 lines · Bruce Richardson
Quoted 42 lines · Bruce Richardson
Quoted 10 lines · David Marchand
Quoted 5 lines · Bruce Richardson

While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's often easier to specify the test name as a commandline parameter to a test binary. This also allows the test runs to be saved as part of the shell cmdline history.

I don't get the argument about history:

Quoted 12 lines
$ history |grep DPDK_TEST
10615  2021-03-24 10:42:11 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10636  2021-03-24 10:51:09 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10653  2021-03-24 11:17:01 ninja-build -C build -j4 &&
DPDK_TEST=kvargs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10794  2021-03-25 18:37:48 history |grep DPDK_TEST

Sure, if you always specify the test name explicitly for each command, rather than running the one test multiple times having set it separately in the environment.
Overall, though I take the point that from a history saving point of view it's a minor saving.

Quoted 11 lines
Quoted 7 lines

This patch adds support for checking all parameters after the EAL ones, and running all valid autotests requested - either from DPDK_TEST or on the commandline. This also allows multiple tests to be run in a single automated session, which is useful for working with components which have multiple test suites.

The same could be achieved splitting DPDK_TEST content with spaces, since test names don't contain one.

Quoted 6 lines
Quoted 5 lines

Yep, that's a useful enhancement too, but I still thing it's better to just have the list of tests appended to the test binary command rather than have to be worrying about properly quoting a specific environment variable at the start of each command.

Ping on this.

I have been using the DPDK_TEST= method for quite some time. But I don't think I ever had a need to run multiple tests since I usually track regressions or hard to reproduce failures. When developing a new test, idem, I used DPDK_TEST=.

Quoted 4 lines

Other instances where using cmdline is preferred over environment

* Calling tests using sudo.

I guess calling a shell with the same command would work, something like:

$ sudo sh -c 'DPDK_TEST=debug_autotest ..../build/app/test/dpdk-test
--no-huge -m 512'
Quoted 1 line

* Calling tests as an execute action when doing a git rebase

I am pretty sure I used DPDK_TEST= in rebase execute actions in the past too. This probably works too:

$ git rebase -i HEAD^ -x 'DPDK_TEST=debug_autotest
build/app/test/dpdk-test --no-huge -m 512'
Quoted 4 lines

Yes, again in both cases, other workarounds are generally available (e.g. sudo -E, and exporting to environemtn before rebase), but also generally less convenient.

You find it less convenient when you want to run multiple tests. I don't have that use case but I don't mind taking it. Can you simply adjust the commitlog, or propose a new wording I can take when applying?

Thanks.

Signature · 1 line
David Marchand
#7 lore raw
Quoted 60 lines · David Marchand
Quoted 44 lines · Bruce Richardson
Quoted 42 lines · Bruce Richardson
Quoted 10 lines · David Marchand
Quoted 5 lines · Bruce Richardson

While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's often easier to specify the test name as a commandline parameter to a test binary. This also allows the test runs to be saved as part of the shell cmdline history.

I don't get the argument about history:

Quoted 12 lines
$ history |grep DPDK_TEST
10615  2021-03-24 10:42:11 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10636  2021-03-24 10:51:09 ninja-build -C build -j4 &&
DPDK_TEST=logs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10653  2021-03-24 11:17:01 ninja-build -C build -j4 &&
DPDK_TEST=kvargs_autotest ./build/app/test/dpdk-test --no-huge -m 512
--log-level=lib.eal:debug
10794  2021-03-25 18:37:48 history |grep DPDK_TEST

Sure, if you always specify the test name explicitly for each command, rather than running the one test multiple times having set it separately in the environment.
Overall, though I take the point that from a history saving point of view it's a minor saving.

Quoted 11 lines
Quoted 7 lines

This patch adds support for checking all parameters after the EAL ones, and running all valid autotests requested - either from DPDK_TEST or on the commandline. This also allows multiple tests to be run in a single automated session, which is useful for working with components which have multiple test suites.

The same could be achieved splitting DPDK_TEST content with spaces, since test names don't contain one.

Quoted 6 lines
Quoted 5 lines

Yep, that's a useful enhancement too, but I still thing it's better to just have the list of tests appended to the test binary command rather than have to be worrying about properly quoting a specific environment variable at the start of each command.

Ping on this.

I have been using the DPDK_TEST= method for quite some time. But I don't think I ever had a need to run multiple tests since I usually track regressions or hard to reproduce failures. When developing a new test, idem, I used DPDK_TEST=.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

Other instances where using cmdline is preferred over environment

* Calling tests using sudo.

I guess calling a shell with the same command would work, something like:

$ sudo sh -c 'DPDK_TEST=debug_autotest ..../build/app/test/dpdk-test
--no-huge -m 512'

Not exactly the simplest command, compared to just
"sudo /path/to/dpdk-test --no-huge -m512 -- debug_autotest"
Using sudo -E is what I tend to prefer in this case. :-)

Quoted 10 lines
Quoted 2 lines

* Calling tests as an execute action when doing a git rebase

I am pretty sure I used DPDK_TEST= in rebase execute actions in the past too. This probably works too:

$ git rebase -i HEAD^ -x 'DPDK_TEST=debug_autotest
build/app/test/dpdk-test --no-huge -m 512'
Quoted 1 line
Quoted 9 lines
Quoted 3 lines

Yes, again in both cases, other workarounds are generally available (e.g. sudo -E, and exporting to environemtn before rebase), but also generally less convenient.

You find it less convenient when you want to run multiple tests. I don't have that use case but I don't mind taking it. Can you simply adjust the commitlog, or propose a new wording I can take when applying?

Would just replacing the first paragraph as below be suitable? Since it's only a single sentence, the second paragraph can also be merged straight in with it to leave a 1-paragraph commit log message.

"While having the ability to run a test based off the DPDK_TEST environment variable is useful, it's sometimes more convenient to specify the test name as a commandline parameter to a test binary."

/Bruce