Make unit tests great again
cover Make unit tests great again
Previously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.
In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.
This patchset conflicts with some of the earlier work on autotests [1] [2] [3], but i think it presents a cleaner solution for some of the problems highlighted by those patch series. I can integrate those patches into this series if need be.
[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://dpdk.org/dev/patchwork/patch/40371/
[3] http://dpdk.org/dev/patchwork/patch/40372/
Anatoly Burakov (7):
autotest: fix printing
autotest: fix invalid code on reports
autotest: make autotest runner python 2/3 compliant
autotest: visually separate different test categories
autotest: improve filtering
autotest: remove autotest grouping
autotest: properly parallelize unit tests
test/test/autotest.py | 13 +- test/test/autotest_data.py | 749 ++++++++++++++--------------------- test/test/autotest_runner.py | 519 ++++++++++++------------ 3 files changed, 586 insertions(+), 695 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/7 autotest: fix printing
Previously, printing was done using tuple syntax, which caused output to appear as a tuple as opposed to being one string. Fix this by using addition operator instead.
--- test/test/autotest_runner.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
test/test/autotest_runner.py +3 -3
@@ -247,7 +247,7 @@ def __process_results(self, results): # don't print out total time every line, it's the same anyway if i == len(results) - 1: - print(result, + print(result + "[%02dm %02ds]" % (total_time / 60, total_time % 60)) else: print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self): # create table header print("") - print("Test name".ljust(30), "Test result".ljust(29), - "Test".center(9), "Total".center(9)) + print("Test name".ljust(30) + "Test result".ljust(29) + + "Test".center(9) + "Total".center(9)) print("=" * 80) # make a note of tests start time
Signature · 1 line
2.17.1
PATCH 2/7 autotest: fix invalid code on reports
In reply to Anatoly BurakovThere are no reports defined for any test, so this codepath was never triggered, but it's still wrong because it's referencing variables that aren't there. Fix it by passing target into the test function, and reference correct log variable.
--- test/test/autotest_runner.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
test/test/autotest_runner.py +7 -5
@@ -41,7 +41,7 @@ def wait_prompt(child): # quite a bit of effort to make it work). -def run_test_group(cmdline, test_group): +def run_test_group(cmdline, target, test_group): results = [] child = None start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group): # make a note when the test was finished end_time = time.time() + log = logfile.getvalue() + # append test data to the result tuple - result += (test["Name"], end_time - start_time, - logfile.getvalue()) + result += (test["Name"], end_time - start_time, log) # call report function, if any defined, and supply it with # target and complete log for test run if test["Report"]: - report = test["Report"](self.target, log) + report = test["Report"](target, log) # append report to results tuple result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self): for test_group in self.parallel_test_groups: result = pool.apply_async(run_test_group, [self.__get_cmdline(test_group), + self.target, test_group]) results.append(result)
@@ -367,7 +369,7 @@ def run_all_tests(self): # run non_parallel tests. they are run one by one, synchronously for test_group in self.non_parallel_test_groups: group_result = run_test_group( - self.__get_cmdline(test_group), test_group) + self.__get_cmdline(test_group), self.target, test_group) self.__process_results(group_result)
Signature · 1 line
2.17.1
PATCH 3/7 autotest: make autotest runner python 2/3 compliant
In reply to Anatoly BurakovAutotest runner was still using python 2-style print syntax. Fix it by importing print function from the future, and fix the calls to be python-3 style.
--- test/test/autotest_runner.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
test/test/autotest_runner.py +4 -3
@@ -3,6 +3,7 @@ # The main logic behind running autotests in parallel +from __future__ import print_function import StringIO import csv import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group): # prepare logging of init startuplog = StringIO.StringIO() - print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"]) - print >>startuplog, "\ncmdline=%s" % cmdline + print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog) + print("\ncmdline=%s" % cmdline, file=startuplog) child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group): try: # print test name to log buffer - print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"]) + print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test if stripped or test["Command"] in avail_cmds:
Signature · 1 line
2.17.1
PATCH 4/7 autotest: visually separate different test categories
In reply to Anatoly BurakovHelp visually identify parallel vs. non-parallel autotests.
--- test/test/autotest_runner.py | 2 ++ 1 file changed, 2 insertions(+)
test/test/autotest_runner.py +2 -0
@@ -341,6 +341,7 @@ def run_all_tests(self): # make a note of tests start time self.start = time.time() + print("Parallel autotests:") # assign worker threads to run test groups for test_group in self.parallel_test_groups: result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self): # remove result from results list once we're done with it results.remove(group_result) + print("Non-parallel autotests:") # run non_parallel tests. they are run one by one, synchronously for test_group in self.non_parallel_test_groups: group_result = run_test_group(
Signature · 1 line
2.17.1
PATCH 5/7 autotest: improve filtering
In reply to Anatoly BurakovImprove code for filtering test groups. Also, move reading binary symbols into filtering stage, so that tests that are meant to be skipped are never attempted to be executed in the first place. Before running tests, print out any tests that were skipped because they weren't compiled.
--- test/test/autotest_runner.py | 118 ++++++++++++++++++++--------------- 1 file changed, 66 insertions(+), 52 deletions(-)
test/test/autotest_runner.py +66 -52
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group): results.append((0, "Success", "Start %s" % test_group["Prefix"], time.time() - start_time, startuplog.getvalue(), None)) - # parse the binary for available test commands - binary = cmdline.split()[0] - stripped = 'not stripped' not in subprocess.check_output(['file', binary]) - if not stripped: - symbols = subprocess.check_output(['nm', binary]).decode('utf-8') - avail_cmds = re.findall('test_register_(\w+)', symbols) - # run all tests in test group for test in test_group["Tests"]:
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group): print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test - if stripped or test["Command"] in avail_cmds: - result = test["Func"](child, test["Command"]) - else: - result = (0, "Skipped [Not Available]") + result = test["Func"](child, test["Command"]) # make a note when the test was finished end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner: def __init__(self, cmdline, target, blacklist, whitelist): self.cmdline = cmdline self.target = target + self.binary = cmdline.split()[0] self.blacklist = blacklist self.whitelist = whitelist + self.skipped = [] # log file filename logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results): if i != 0: self.csvwriter.writerow([test_name, test_result, result_str]) - # this function iterates over test groups and removes each - # test that is not in whitelist/blacklist - def __filter_groups(self, test_groups): - groups_to_remove = [] - - # filter out tests from parallel test groups - for i, test_group in enumerate(test_groups): - - # iterate over a copy so that we could safely delete individual - # tests - for test in test_group["Tests"][:]: - test_id = test["Command"] - - # dump tests are specified in full e.g. "Dump_mempool" - if "_autotest" in test_id: - test_id = test_id[:-len("_autotest")] - - # filter out blacklisted/whitelisted tests - if self.blacklist and test_id in self.blacklist: - test_group["Tests"].remove(test) - continue - if self.whitelist and test_id not in self.whitelist: - test_group["Tests"].remove(test) - continue - - # modify or remove original group - if len(test_group["Tests"]) > 0: - test_groups[i] = test_group - else: - # remember which groups should be deleted - # put the numbers backwards so that we start - # deleting from the end, not from the beginning - groups_to_remove.insert(0, i) + # this function checks individual test and decides if this test should be in + # the group by comparing it against whitelist/blacklist. it also checks if + # the test is compiled into the binary, and marks it as skipped if necessary + def __filter_test(self, test): + test_cmd = test["Command"] + test_id = test_cmd + + # dump tests are specified in full e.g. "Dump_mempool" + if "_autotest" in test_id: + test_id = test_id[:-len("_autotest")] + + # filter out blacklisted/whitelisted tests + if self.blacklist and test_id in self.blacklist: + return False + if self.whitelist and test_id not in self.whitelist: + return False + + # if test wasn't compiled in, remove it as well + + # parse the binary for available test commands + stripped = 'not stripped' not in \ + subprocess.check_output(['file', self.binary]) + if not stripped: + symbols = subprocess.check_output(['nm', + self.binary]).decode('utf-8') + avail_cmds = re.findall('test_register_(\w+)', symbols) + + if test_cmd not in avail_cmds: + # notify user + result = 0, "Skipped [Not compiled]", test_id, 0, "", None + self.skipped.append(tuple(result)) + return False - # remove test groups that need to be removed - for i in groups_to_remove: - del test_groups[i] + return True - return test_groups + def __filter_group(self, group): + group["Tests"] = list(filter(self.__filter_test, group["Tests"])) + return len(group["Tests"]) > 0 # iterate over test groups and run tests associated with them def run_all_tests(self): # filter groups - self.parallel_test_groups = \ - self.__filter_groups(self.parallel_test_groups) - self.non_parallel_test_groups = \ - self.__filter_groups(self.non_parallel_test_groups) + # for each test group, check all tests against the filter, then remove + # all groups that don't have any tests + self.parallel_test_groups = list( + filter(self.__filter_group, + self.parallel_test_groups) + ) + self.non_parallel_test_groups = list( + filter(self.__filter_group, + self.non_parallel_test_groups) + ) # create a pool of worker threads pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self): "Test".center(9) + "Total".center(9)) print("=" * 80) + # print out skipped autotests if there were any + if len(self.skipped): + print("Skipped autotests:") + + # print out any skipped tests + for result in self.skipped: + # unpack result tuple + test_result, result_str, test_name, _, _, _ = result + self.csvwriter.writerow([test_name, test_result, + result_str]) + + t = ("%s:" % test_name).ljust(30) + t += result_str.ljust(29) + t += "[00m 00s]" + + print(t) + # make a note of tests start time self.start = time.time()
Signature · 1 line
2.17.1
PATCH 6/7 autotest: remove autotest grouping
In reply to Anatoly BurakovPreviously, all autotests were grouped into (seemingly arbitrary) groups. The goal was to run all tests in parallel (so that autotest finishes faster), but we couldn't just do it willy-nilly because DPDK couldn't allocate and free hugepages on-demand, so we had to find autotest groupings that could work memory-wise and still be fast enough to not hold up shorter tests. The inflexibility of memory subsystem has now been fixed for 18.05, so grouping autotests is no longer necessary.
Thus, this commit moves all autotests into two groups - parallel(izable) autotests, and non-arallel(izable) autotests (typically performance tests). Note that this particular commit makes running autotests dog slow because while the tests are now in a single group, the test function itself hasn't changed much, so all autotests are now run one-by-one, starting and stopping the DPDK test application.
--- test/test/autotest.py | 7 +- test/test/autotest_data.py | 749 ++++++++++++++--------------------- test/test/autotest_runner.py | 271 +++++-------- 3 files changed, 408 insertions(+), 619 deletions(-)
test/test/autotest.py +2 -5
@@ -39,11 +39,8 @@ def usage(): runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist, test_whitelist) -for test_group in autotest_data.parallel_test_group_list: - runner.add_parallel_test_group(test_group) - -for test_group in autotest_data.non_parallel_test_group_list: - runner.add_non_parallel_test_group(test_group) +runner.parallel_tests = autotest_data.parallel_test_list[:] +runner.non_parallel_tests = autotest_data.non_parallel_test_list[:] num_fails = runner.run_all_tests()
test/test/autotest_data.py +303 -446
768 lines not on this page · raw message
test/test/autotest_runner.py +103 -168
356 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH 7/7 autotest: properly parallelize unit tests
In reply to Anatoly BurakovNow that everything else is in place, we can run unit tests in a different fashion to what they were running as before. Previously, we had all autotests as part of groups (largely obtained through trial and error) to ensure parallel execution while still limiting amounts of memory used by those tests.
This is no longer necessary, and as of previous commit, all tests are now in the same group (still broken into two categories). They still run one-by-one though. Fix this by initializing child processes in multiprocessing Pool initialization, and putting all tests on the queue, so that tests are executed by the first idle worker. Tests are also affinitized to different NUMA nodes using taskset in a round-robin fashion, to prevent over-exhausting memory on any given NUMA node.
Non-parallel tests are executed in similar fashion, but on a separate queue which will have only one pool worker, ensuring non-parallel execution.
Support for FreeBSD is also added to ensure that on FreeBSD, all tests are run sequentially even for the parallel section.
--- test/test/autotest.py | 6 +- test/test/autotest_runner.py | 277 ++++++++++++++++++++++------------- 2 files changed, 183 insertions(+), 100 deletions(-)
test/test/autotest.py +5 -1
@@ -36,8 +36,12 @@ def usage(): print(cmdline) +# how many workers to run tests with. FreeBSD doesn't support multiple primary +# processes, so make it 1, otherwise make it 4. ignored for non-parallel tests +n_processes = 1 if "bsdapp" in target else 4 + runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist, - test_whitelist) + test_whitelist, n_processes) runner.parallel_tests = autotest_data.parallel_test_list[:] runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
test/test/autotest_runner.py +178 -99
369 lines not on this page · raw message
Signature · 1 line
2.17.1
+Cc Jananee
07/06/2018 23:01, Anatoly Burakov:
Quoted 18 lines
Previously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.
In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.
This patchset conflicts with some of the earlier work on autotests [1] [2] [3], but i think it presents a cleaner solution for some of the problems highlighted by those patch series. I can integrate those patches into this series if need be.
Quoted 3 lines
[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://dpdk.org/dev/patchwork/patch/40371/
[3] http://dpdk.org/dev/patchwork/patch/40372/
It may be interesting to work on lists of tests as done in the following patch:
http://dpdk.org/dev/patchwork/patch/40373/
The idea is to split tests in several categories:
- basic and short test
- longer and lower priority
- performance test
As a long term solution, we can think about making category an attribute inside the test itself?
Quoted 27 lines · Thomas Monjalon
+Cc Jananee
07/06/2018 23:01, Anatoly Burakov:
Quoted 18 lines
Previously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.
In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.
This patchset conflicts with some of the earlier work on autotests [1] [2] [3], but i think it presents a cleaner solution for some of the problems highlighted by those patch series. I can integrate those patches into this series if need be.
Quoted 3 lines
[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://dpdk.org/dev/patchwork/patch/40371/
[3] http://dpdk.org/dev/patchwork/patch/40372/It may be interesting to work on lists of tests as done in the following patch:
Quoted 9 lines
http://dpdk.org/dev/patchwork/patch/40373/The idea is to split tests in several categories:
- basic and short test
- longer and lower priority
- performance test
As a long term solution, we can think about making category an attribute inside the test itself?
These test categories do not conflict with my patchset as they ultimately rely on white/blacklisting, which will continue to work as before.
In my view, it really boils down to two things - either tests can be run in parallel with others (i.e. their result won't be affected by another independent DPDK test app instance), or not. On top of that, we can use blacklisting or whitelisting to define which tests will actually be run (i.e. define any "categories" we want), but their (non-)parallelism must always be respected to get good test results.
Signature · 2 lines
Thanks, Anatoly
Quoted 41 lines · Anatoly
Quoted 27 lines · Thomas Monjalon
+Cc Jananee
07/06/2018 23:01, Anatoly Burakov:
Quoted 18 lines
Previously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.
In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.
This patchset conflicts with some of the earlier work on autotests [1] [2] [3], but i think it presents a cleaner solution for some of the problems highlighted by those patch series. I can integrate those patches into this series if need be.
Quoted 3 lines
[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://dpdk.org/dev/patchwork/patch/40371/
[3] http://dpdk.org/dev/patchwork/patch/40372/It may be interesting to work on lists of tests as done in the following patch:
Quoted 9 lines
http://dpdk.org/dev/patchwork/patch/40373/The idea is to split tests in several categories:
- basic and short test
- longer and lower priority
- performance test
As a long term solution, we can think about making category an attribute inside the test itself?These test categories do not conflict with my patchset as they ultimately rely on white/blacklisting, which will continue to work as before.
Quoted 7 lines
In my view, it really boils down to two things - either tests can be run in parallel with others (i.e. their result won't be affected by another independent DPDK test app instance), or not. On top of that, we can use blacklisting or whitelisting to define which tests will actually be run (i.e. define any "categories" we want), but their (non-)parallelism must always be respected to get good test results.
Have you looked at: http://mesonbuild.com/Unit-tests.html, at it would be good to transition away from our own custom script infrastructure for the tests.
There is already some support for running the unit tests using "meson test", but it could do with some more cleanup e.g. to move the tests into suitable suites (corresponding to the categories Thomas has suggested). We could also do with specifying properly what tests are parallel-safe and what aren't.
PATCH v2 1/10 autotest: fix printing
In reply to Anatoly BurakovPreviously, printing was done using tuple syntax, which caused output to appear as a tuple as opposed to being one string. Fix this by using addition operator instead.
--- test/test/autotest_runner.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
test/test/autotest_runner.py +3 -3
@@ -247,7 +247,7 @@ def __process_results(self, results): # don't print out total time every line, it's the same anyway if i == len(results) - 1: - print(result, + print(result + "[%02dm %02ds]" % (total_time / 60, total_time % 60)) else: print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self): # create table header print("") - print("Test name".ljust(30), "Test result".ljust(29), - "Test".center(9), "Total".center(9)) + print("Test name".ljust(30) + "Test result".ljust(29) + + "Test".center(9) + "Total".center(9)) print("=" * 80) # make a note of tests start time
Signature · 1 line
2.14.4
PATCH v2 2/10 autotest: fix invalid code on reports
In reply to Anatoly BurakovThere are no reports defined for any test, so this codepath was never triggered, but it's still wrong because it's referencing variables that aren't there. Fix it by passing target into the test function, and reference correct log variable.
--- test/test/autotest_runner.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
test/test/autotest_runner.py +7 -5
@@ -41,7 +41,7 @@ def wait_prompt(child): # quite a bit of effort to make it work). -def run_test_group(cmdline, test_group): +def run_test_group(cmdline, target, test_group): results = [] child = None start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group): # make a note when the test was finished end_time = time.time() + log = logfile.getvalue() + # append test data to the result tuple - result += (test["Name"], end_time - start_time, - logfile.getvalue()) + result += (test["Name"], end_time - start_time, log) # call report function, if any defined, and supply it with # target and complete log for test run if test["Report"]: - report = test["Report"](self.target, log) + report = test["Report"](target, log) # append report to results tuple result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self): for test_group in self.parallel_test_groups: result = pool.apply_async(run_test_group, [self.__get_cmdline(test_group), + self.target, test_group]) results.append(result)
@@ -367,7 +369,7 @@ def run_all_tests(self): # run non_parallel tests. they are run one by one, synchronously for test_group in self.non_parallel_test_groups: group_result = run_test_group( - self.__get_cmdline(test_group), test_group) + self.__get_cmdline(test_group), self.target, test_group) self.__process_results(group_result)
Signature · 1 line
2.14.4
PATCH v2 3/10 autotest: make autotest runner python 2/3 compliant
In reply to Anatoly BurakovAutotest runner was still using python 2-style print syntax. Fix it by importing print function from the future, and fix the calls to be python-3 style.
--- test/test/autotest_runner.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
test/test/autotest_runner.py +4 -3
@@ -3,6 +3,7 @@ # The main logic behind running autotests in parallel +from __future__ import print_function import StringIO import csv import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group): # prepare logging of init startuplog = StringIO.StringIO() - print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"]) - print >>startuplog, "\ncmdline=%s" % cmdline + print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog) + print("\ncmdline=%s" % cmdline, file=startuplog) child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group): try: # print test name to log buffer - print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"]) + print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test if stripped or test["Command"] in avail_cmds:
Signature · 1 line
2.14.4
PATCH v2 4/10 autotest: visually separate different test categories
In reply to Anatoly BurakovHelp visually identify parallel vs. non-parallel autotests.
--- test/test/autotest_runner.py | 2 ++ 1 file changed, 2 insertions(+)
test/test/autotest_runner.py +2 -0
@@ -341,6 +341,7 @@ def run_all_tests(self): # make a note of tests start time self.start = time.time() + print("Parallel autotests:") # assign worker threads to run test groups for test_group in self.parallel_test_groups: result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self): # remove result from results list once we're done with it results.remove(group_result) + print("Non-parallel autotests:") # run non_parallel tests. they are run one by one, synchronously for test_group in self.non_parallel_test_groups: group_result = run_test_group(
Signature · 1 line
2.14.4
PATCH v2 5/10 autotest: improve filtering
In reply to Anatoly BurakovImprove code for filtering test groups. Also, move reading binary symbols into filtering stage, so that tests that are meant to be skipped are never attempted to be executed in the first place. Before running tests, print out any tests that were skipped because they weren't compiled.
--- test/test/autotest_runner.py | 118 ++++++++++++++++++++++++------------------- 1 file changed, 66 insertions(+), 52 deletions(-)
test/test/autotest_runner.py +66 -52
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group): results.append((0, "Success", "Start %s" % test_group["Prefix"], time.time() - start_time, startuplog.getvalue(), None)) - # parse the binary for available test commands - binary = cmdline.split()[0] - stripped = 'not stripped' not in subprocess.check_output(['file', binary]) - if not stripped: - symbols = subprocess.check_output(['nm', binary]).decode('utf-8') - avail_cmds = re.findall('test_register_(\w+)', symbols) - # run all tests in test group for test in test_group["Tests"]:
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group): print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test - if stripped or test["Command"] in avail_cmds: - result = test["Func"](child, test["Command"]) - else: - result = (0, "Skipped [Not Available]") + result = test["Func"](child, test["Command"]) # make a note when the test was finished end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner: def __init__(self, cmdline, target, blacklist, whitelist): self.cmdline = cmdline self.target = target + self.binary = cmdline.split()[0] self.blacklist = blacklist self.whitelist = whitelist + self.skipped = [] # log file filename logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results): if i != 0: self.csvwriter.writerow([test_name, test_result, result_str]) - # this function iterates over test groups and removes each - # test that is not in whitelist/blacklist - def __filter_groups(self, test_groups): - groups_to_remove = [] - - # filter out tests from parallel test groups - for i, test_group in enumerate(test_groups): - - # iterate over a copy so that we could safely delete individual - # tests - for test in test_group["Tests"][:]: - test_id = test["Command"] - - # dump tests are specified in full e.g. "Dump_mempool" - if "_autotest" in test_id: - test_id = test_id[:-len("_autotest")] - - # filter out blacklisted/whitelisted tests - if self.blacklist and test_id in self.blacklist: - test_group["Tests"].remove(test) - continue - if self.whitelist and test_id not in self.whitelist: - test_group["Tests"].remove(test) - continue - - # modify or remove original group - if len(test_group["Tests"]) > 0: - test_groups[i] = test_group - else: - # remember which groups should be deleted - # put the numbers backwards so that we start - # deleting from the end, not from the beginning - groups_to_remove.insert(0, i) + # this function checks individual test and decides if this test should be in + # the group by comparing it against whitelist/blacklist. it also checks if + # the test is compiled into the binary, and marks it as skipped if necessary + def __filter_test(self, test): + test_cmd = test["Command"] + test_id = test_cmd + + # dump tests are specified in full e.g. "Dump_mempool" + if "_autotest" in test_id: + test_id = test_id[:-len("_autotest")] + + # filter out blacklisted/whitelisted tests + if self.blacklist and test_id in self.blacklist: + return False + if self.whitelist and test_id not in self.whitelist: + return False + + # if test wasn't compiled in, remove it as well + + # parse the binary for available test commands + stripped = 'not stripped' not in \ + subprocess.check_output(['file', self.binary]) + if not stripped: + symbols = subprocess.check_output(['nm', + self.binary]).decode('utf-8') + avail_cmds = re.findall('test_register_(\w+)', symbols) + + if test_cmd not in avail_cmds: + # notify user + result = 0, "Skipped [Not compiled]", test_id, 0, "", None + self.skipped.append(tuple(result)) + return False - # remove test groups that need to be removed - for i in groups_to_remove: - del test_groups[i] + return True - return test_groups + def __filter_group(self, group): + group["Tests"] = list(filter(self.__filter_test, group["Tests"])) + return len(group["Tests"]) > 0 # iterate over test groups and run tests associated with them def run_all_tests(self): # filter groups - self.parallel_test_groups = \ - self.__filter_groups(self.parallel_test_groups) - self.non_parallel_test_groups = \ - self.__filter_groups(self.non_parallel_test_groups) + # for each test group, check all tests against the filter, then remove + # all groups that don't have any tests + self.parallel_test_groups = list( + filter(self.__filter_group, + self.parallel_test_groups) + ) + self.non_parallel_test_groups = list( + filter(self.__filter_group, + self.non_parallel_test_groups) + ) # create a pool of worker threads pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self): "Test".center(9) + "Total".center(9)) print("=" * 80) + # print out skipped autotests if there were any + if len(self.skipped): + print("Skipped autotests:") + + # print out any skipped tests + for result in self.skipped: + # unpack result tuple + test_result, result_str, test_name, _, _, _ = result + self.csvwriter.writerow([test_name, test_result, + result_str]) + + t = ("%s:" % test_name).ljust(30) + t += result_str.ljust(29) + t += "[00m 00s]" + + print(t) + # make a note of tests start time self.start = time.time()
Signature · 1 line
2.14.4
PATCH v2 6/10 autotest: remove autotest grouping
In reply to Anatoly BurakovPreviously, all autotests were grouped into (seemingly arbitrary) groups. The goal was to run all tests in parallel (so that autotest finishes faster), but we couldn't just do it willy-nilly because DPDK couldn't allocate and free hugepages on-demand, so we had to find autotest groupings that could work memory-wise and still be fast enough to not hold up shorter tests. The inflexibility of memory subsystem has now been fixed for 18.05, so grouping autotests is no longer necessary.
Thus, this commit moves all autotests into two groups - parallel(izable) autotests, and non-arallel(izable) autotests (typically performance tests). Note that this particular commit makes running autotests dog slow because while the tests are now in a single group, the test function itself hasn't changed much, so all autotests are now run one-by-one, starting and stopping the DPDK test application.
--- test/test/autotest.py | 7 +- test/test/autotest_data.py | 749 +++++++++++++++++-------------------------- test/test/autotest_runner.py | 271 ++++++---------- 3 files changed, 408 insertions(+), 619 deletions(-)
test/test/autotest.py +2 -5
@@ -39,11 +39,8 @@ def usage(): runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist, test_whitelist) -for test_group in autotest_data.parallel_test_group_list: - runner.add_parallel_test_group(test_group) - -for test_group in autotest_data.non_parallel_test_group_list: - runner.add_non_parallel_test_group(test_group) +runner.parallel_tests = autotest_data.parallel_test_list[:] +runner.non_parallel_tests = autotest_data.non_parallel_test_list[:] num_fails = runner.run_all_tests()
test/test/autotest_data.py +303 -446
768 lines not on this page · raw message
test/test/autotest_runner.py +103 -168
356 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH v2 7/10 autotest: properly parallelize unit tests
In reply to Anatoly BurakovNow that everything else is in place, we can run unit tests in a different fashion to what they were running as before. Previously, we had all autotests as part of groups (largely obtained through trial and error) to ensure parallel execution while still limiting amounts of memory used by those tests.
This is no longer necessary, and as of previous commit, all tests are now in the same group (still broken into two categories). They still run one-by-one though. Fix this by initializing child processes in multiprocessing Pool initialization, and putting all tests on the queue, so that tests are executed by the first idle worker. Tests are also affinitized to different NUMA nodes using taskset in a round-robin fashion, to prevent over-exhausting memory on any given NUMA node.
Non-parallel tests are executed in similar fashion, but on a separate queue which will have only one pool worker, ensuring non-parallel execution.
Support for FreeBSD is also added to ensure that on FreeBSD, all tests are run sequentially even for the parallel section.
--- test/test/autotest.py | 6 +- test/test/autotest_runner.py | 277 +++++++++++++++++++++++++++---------------- 2 files changed, 183 insertions(+), 100 deletions(-)
test/test/autotest.py +5 -1
@@ -36,8 +36,12 @@ def usage(): print(cmdline) +# how many workers to run tests with. FreeBSD doesn't support multiple primary +# processes, so make it 1, otherwise make it 4. ignored for non-parallel tests +n_processes = 1 if "bsdapp" in target else 4 + runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist, - test_whitelist) + test_whitelist, n_processes) runner.parallel_tests = autotest_data.parallel_test_list[:] runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
test/test/autotest_runner.py +178 -99
369 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH v2 8/10 autotest: add new test cases to autotest
In reply to Anatoly BurakovAutotest is enhanced with additional test cases
being added to autotest_data.py
--- test/test/autotest_data.py | 370 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 362 insertions(+), 8 deletions(-)
test/test/autotest_data.py +362 -8
@@ -134,12 +134,6 @@ "Func": default_autotest, "Report": None, }, - { - "Name": "PCI autotest", - "Command": "pci_autotest", - "Func": default_autotest, - "Report": None, - }, { "Name": "Malloc autotest", "Command": "malloc_autotest",
@@ -248,6 +242,311 @@ "Func": default_autotest, "Report": None, }, + { + "Name": "Eventdev selftest octeontx", + "Command": "eventdev_selftest_octeontx", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Event ring autotest", + "Command": "event_ring_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Table autotest", + "Command": "table_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Flow classify autotest", + "Command": "flow_classify_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump struct sizes", + "Command": "dump_struct_sizes", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump mempool", + "Command": "dump_mempool", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump malloc stats", + "Command": "dump_malloc_stats", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump devargs", + "Command": "dump_devargs", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump log types", + "Command": "dump_log_types", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump_ring", + "Command": "dump_ring", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Quit", + "Command": "quit", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx mode", + "Command": "set_rxtx_mode", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx anchor", + "Command": "set_rxtx_anchor", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx sc", + "Command": "set_rxtx_sc", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Event eth rx adapter autotest", + "Command": "event_eth_rx_adapter_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Rawdev autotest", + "Command": "rawdev_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Kvargs autotest", + "Command": "kvargs_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump physmem", + "Command": "dump_physmem", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump memzone", + "Command": "dump_memzone", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Devargs autotest", + "Command": "devargs_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding autotest", + "Command": "link_bonding_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding mode4 autotest", + "Command": "link_bonding_mode4_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding rssconf autotest", + "Command": "link_bonding_rssconf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Crc autotest", + "Command": "crc_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Distributor autotest", + "Command": "distributor_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reorder autotest", + "Command": "reorder_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Barrier autotest", + "Command": "barrier_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "User delay", + "Command": "user_delay_us", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Bitmap test", + "Command": "bitmap_test", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash scaling autotest", + "Command": "hash_scaling_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash multiwriter autotest", + "Command": "hash_multiwriter_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Service autotest", + "Command": "service_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Timer racecond autotest", + "Command": "timer_racecond_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Member autotest", + "Command": "member_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Efd_autotest", + "Command": "efd_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Thash autotest", + "Command": "thash_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash function autotest", + "Command": "hash_functions_autotest", + "Func": default_autotest, + "Report": None, + }, + "Name": "Cryptodev sw mrvl autotest", + "Command": "cryptodev_sw_mrvl_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev dpaa2 sec autotest", + "Command": "cryptodev_dpaa2_sec_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev dpaa sec autotest", + "Command": "cryptodev_dpaa_sec_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev qat autotest", + "Command": "cryptodev_qat_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev aesni mb autotest", + "Command": "cryptodev_aesni_mb_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev openssl autotest", + "Command": "cryptodev_openssl_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev scheduler autotest", + "Command": "cryptodev_scheduler_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev aesni gcm autotest", + "Command": "cryptodev_aesni_gcm_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev null autotest", + "Command": "cryptodev_null_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev sw snow3g autotest", + "Command": "cryptodev_sw_snow3g_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev sw kasumi autotest", + "Command": "cryptodev_sw_kasumi_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev_sw_zuc_autotest", + "Command": "cryptodev_sw_zuc_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reciprocal division", + "Command": "reciprocal_division", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Red all", + "Command": "red_all", + "Func": default_autotest, + "Report": None, + }, ] # tests that should not be run when any other tests are running
@@ -259,8 +558,8 @@ "Report": None, }, { - "Name": "Eventdev sw autotest", - "Command": "eventdev_sw_autotest", + "Name": "Eventdev selftest sw", + "Command": "eventdev_selftest_sw", "Func": default_autotest, "Report": None, },
@@ -312,6 +611,61 @@ "Func": default_autotest, "Report": None, }, + { + + "Name": "Pmd perf autotest", + "Command": "pmd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Ring pmd perf autotest", + "Command": "ring_pmd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Distributor perf autotest", + "Command": "distributor_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Red_perf", + "Command": "red_perf", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Lpm6 perf autotest", + "Command": "lpm6_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Lpm perf autotest", + "Command": "lpm_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Efd perf autotest", + "Command": "efd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Member perf autotest", + "Command": "member_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reciprocal division perf", + "Command": "reciprocal_division_perf", + "Func": default_autotest, + "Report": None, + }, # # Please always make sure that ring_perf is the last test! #
Signature · 1 line
2.14.4
Re: [PATCH v2 08/10] autotest: add new test cases to autotest
Quoted 2 lines · Reshma Pattan
Autotest is enhanced with additional test cases
being added toautotest_data.py
You're also removing PCI autotest - commit message needs to call this out as well.
Quoted 10 lines
Signed-off-by Reshma Pattan <[email protected]>Signed-off-by Jananee Parthasarathy <[email protected]>--- test/test/autotest_data.py | 370 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 362 insertions(+), 8 deletions(-)test/test/autotest_data.py +0
-0
<snip>
test/test/autotest_data.py line 310
Quoted 12 lines
test/test/autotest_data.py +12
-0+ { + "Name": "Dump_ring", + "Command": "dump_ring", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Quit", + "Command": "quit", + "Func": default_autotest, + "Report": None, + },
Why is "quit" an autotest? If you want to test "quit" functionality, i would suggest putting it at the end of non-parallel tests, to avoid situation where test prematurely stops (for example, this will happen on FreeBSD, where there's only one thread executing these tests).
But really, having it in this list is IMO dubious.
Quoted 38 lines
test/test/autotest_data.py +38
-0+ { + "Name": "Set rxtx mode", + "Command": "set_rxtx_mode", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx anchor", + "Command": "set_rxtx_anchor", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx sc", + "Command": "set_rxtx_sc", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Event eth rx adapter autotest", + "Command": "event_eth_rx_adapter_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Rawdev autotest", + "Command": "rawdev_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Kvargs autotest", + "Command": "kvargs_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump physmem",
test/test/autotest_data.py line 358
Quoted 10 lines
test/test/autotest_data.py +10
-0+ "Command": "dump_physmem", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Dump memzone", + "Command": "dump_memzone", + "Func": default_autotest, + "Report": None, + },
Here and in other "dump" autotests - do these dump tests work as autotests? Default autotest function expects a "Test OK" at the end of test, and i don't think dump autotests return that, so this will (should?) cause a very long timeout and a test failure. These should have "dump_autotest" as "Func" value - that function correctly parses output of dump autotests (or, to be more accurate, it doesn't - as long as test doesn't crash, it's considered to be successful :) ).
Also, for readability, we should move these dump tests together and have all dump tests one after the other.
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v2 08/10] autotest: add new test cases to autotest
Hi,
Quoted 9 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Friday, July 13, 2018 5:41 PM
To: Pattan, Reshma <[email protected]>; [email protected]; [email protected]
Cc: Parthasarathy, JananeeX M <[email protected]>; [email protected]
Subject: Re: [PATCH v2 08/10] autotest: add new test cases to autotest
Quoted 7 lines
Quoted 2 lines · Reshma Pattan
Autotest is enhanced with additional test cases being added to
autotest_data.pyYou're also removing PCI autotest - commit message needs to call this out as well.
OK
Quoted 15 lines
<snip>
test/test/autotest_data.py line 310
Quoted 12 lines
test/test/autotest_data.py +12
-0+ { + "Name": "Dump_ring", + "Command": "dump_ring", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Quit", + "Command": "quit", + "Func": default_autotest, + "Report": None, + },
Quoted 7 lines
Why is "quit" an autotest? If you want to test "quit" functionality, i would suggest putting it at the end of non-parallel tests, to avoid situation where test prematurely stops (for example, this will happen on FreeBSD, where there's only one thread executing these tests).
But really, having it in this list is IMO dubious.
Ok, I will remove this
Quoted 11 lines
Here and in other "dump" autotests - do these dump tests work as autotests? Default autotest function expects a "Test OK" at the end of test, and i don't think dump autotests return that, so this will
(should?) cause a very long timeout and a test failure. These should have "dump_autotest" as "Func" value - that function correctly parses output of dump autotests (or, to be more accurate, it doesn't - as long as test doesn't crash, it's considered to be successful :) ).Also, for readability, we should move these dump tests together and have all dump tests one after the other.
Ok will add "Func" value as dump_autotest and move all of the dump tests to end of parallel list.
v2 cover Make unit tests great again
In reply to Anatoly BurakovPreviously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.
In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.
This patchset merges changes done in [1], [2] and [3]
[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://dpdk.org/dev/patchwork/patch/40371/
[3] http://patches.dpdk.org/patch/40373/
v2: merged changes done in patch set [1],[2] and [3] and created patches 08/10, 09/10, 10/10
Anatoly Burakov (7):
autotest: fix printing
autotest: fix invalid code on reports
autotest: make autotest runner python 2/3 compliant
autotest: visually separate different test categories
autotest: improve filtering
autotest: remove autotest grouping
autotest: properly parallelize unit tests
Jananee Parthasarathy (1):
mk: update make targets for classified testcases
Reshma Pattan (2):
autotest: add new test cases to autotest
autotest: update result for skipped test cases
mk/rte.sdkroot.mk | 4 +- mk/rte.sdktest.mk | 33 +- test/test/autotest.py | 13 +- test/test/autotest_data.py | 1101 +++++++++++++++++++++++--------------- test/test/autotest_runner.py | 519 +++++++++--------- test/test/autotest_test_funcs.py | 6 +- 6 files changed, 972 insertions(+), 704 deletions(-)
Signature · 1 line
2.14.4
PATCH v2 9/10 autotest: update result for skipped test cases
In reply to Anatoly BurakovFixed in autotest_test_funcs.py to handle test cases which returns "Skipped" as result.
The issue was skipped test cases got timed out, causing delay in autotests execution.
--- test/test/autotest_test_funcs.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
test/test/autotest_test_funcs.py +4 -2
@@ -1,5 +1,5 @@ # SPDX-License-Identifier: BSD-3-Clause -# Copyright(c) 2010-2014 Intel Corporation +# Copyright(c) 2010-2018 Intel Corporation # Test functions
@@ -12,12 +12,14 @@ def default_autotest(child, test_name): child.sendline(test_name) result = child.expect(["Test OK", "Test Failed", - "Command not found", pexpect.TIMEOUT], timeout=900) + "Command not found", "Skipped", pexpect.TIMEOUT], timeout=900) if result == 1: return -1, "Fail" elif result == 2: return -1, "Fail [Not found]" elif result == 3: + return -1, "Fail [Test returns Skipped]" + elif result == 4: return -1, "Fail [Timeout]" return 0, "Success"
Signature · 1 line
2.14.4
Re: [PATCH v2 09/10] autotest: update result for skipped test cases
Quoted 10 lines · Reshma Pattan
Fixed in
autotest_test_funcs.pyto handle test cases which returns "Skipped" as result.
The issue was skipped test cases got timed out, causing delay in autotests execution.Signed-off-by Jananee Parthasarathy <[email protected]>Signed-off-by Reshma Pattan <[email protected]>---
Is this patch still applicable? Patch 5 in these series causes skipped tests to not get executed in the first place, so by the time we reach this patch it is no longer a problem, i think.
Signature · 2 lines
Thanks, Anatoly
PATCH v2 10/10 mk: update make targets for classified testcases
In reply to Anatoly BurakovMakefiles are updated with new test case lists.
Test cases are classified as -
P1 - Main test cases,
P2 - Cryptodev/driver test cases,
P3 - Perf test cases which takes longer than 10s,
P4 - Logging/Dump test cases.
Makefile is updated with different targets
for the above classified groups.
Test cases for different targets are listed accordingly.
--- mk/rte.sdkroot.mk | 4 ++-- mk/rte.sdktest.mk | 33 +++++++++++++++++++++++++++------ 2 files changed, 29 insertions(+), 8 deletions(-)
mk/rte.sdkroot.mk +2 -2
@@ -68,8 +68,8 @@ config defconfig showconfigs showversion showversionum: cscope gtags tags etags: $(Q)$(RTE_SDK)/devtools/build-tags.sh $@ $T -.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage -test test-basic test-fast test-ring test-mempool test-perf coverage: +.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump +test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump: $(Q)$(MAKE) -f $(RTE_SDK)/mk/rte.sdktest.mk $@ test: test-build
mk/rte.sdktest.mk +27 -6
@@ -18,14 +18,35 @@ DIR := $(shell basename $(RTE_OUTPUT)) # # test: launch auto-tests, very simple for now. # -.PHONY: test test-basic test-fast test-perf coverage +.PHONY: test test-basic test-fast test-perf test-drivers test-dump coverage -PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf -coverage: BLACKLIST=-$(PERFLIST) -test-fast: BLACKLIST=-$(PERFLIST) -test-perf: WHITELIST=$(PERFLIST) +PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf,\ + reciprocal_division,reciprocal_division_perf,lpm_perf,red_all,\ + barrier,hash_multiwriter,timer_racecond,efd,hash_functions,\ + eventdev_selftest_sw,member_perf,efd_perf,lpm6_perf,red_perf,\ + distributor_perf,ring_pmd_perf,pmd_perf,ring_perf +DRIVERSLIST=link_bonding,link_bonding_mode4,link_bonding_rssconf,\ + cryptodev_sw_mrvl,cryptodev_dpaa2_sec,cryptodev_dpaa_sec,\ + cryptodev_qat,cryptodev_aesni_mb,cryptodev_openssl,\ + cryptodev_scheduler,cryptodev_aesni_gcm,cryptodev_null,\ + cryptodev_sw_snow3g,cryptodev_sw_kasumi,cryptodev_sw_zuc +DUMPLIST=dump_struct_sizes,dump_mempool,dump_malloc_stats,dump_devargs,\ + dump_log_types,dump_ring,quit,dump_physmem,dump_memzone,\ + devargs_autotest -test test-basic test-fast test-perf: +SPACESTR:= +SPACESTR+= +STRIPPED_PERFLIST=$(subst $(SPACESTR),,$(PERFLIST)) +STRIPPED_DRIVERSLIST=$(subst $(SPACESTR),,$(DRIVERSLIST)) +STRIPPED_DUMPLIST=$(subst $(SPACESTR),,$(DUMPLIST)) + +coverage: BLACKLIST=-$(STRIPPED_PERFLIST) +test-fast: BLACKLIST=-$(STRIPPED_PERFLIST),$(STRIPPED_DRIVERSLIST),$(STRIPPED_DUMPLIST) +test-perf: WHITELIST=$(STRIPPED_PERFLIST) +test-drivers: WHITELIST=$(STRIPPED_DRIVERSLIST) +test-dump: WHITELIST=$(STRIPPED_DUMPLIST) + +test test-basic test-fast test-perf test-drivers test-dump: @mkdir -p $(AUTOTEST_DIR) ; \ cd $(AUTOTEST_DIR) ; \ if [ -f $(RTE_OUTPUT)/app/test ]; then \
Signature · 1 line
2.14.4
PATCH v3 1/9 autotest: fix printing
In reply to Anatoly BurakovPreviously, printing was done using tuple syntax, which caused output to appear as a tuple as opposed to being one string. Fix this by using addition operator instead.
--- test/test/autotest_runner.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
test/test/autotest_runner.py +3 -3
@@ -247,7 +247,7 @@ def __process_results(self, results): # don't print out total time every line, it's the same anyway if i == len(results) - 1: - print(result, + print(result + "[%02dm %02ds]" % (total_time / 60, total_time % 60)) else: print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self): # create table header print("") - print("Test name".ljust(30), "Test result".ljust(29), - "Test".center(9), "Total".center(9)) + print("Test name".ljust(30) + "Test result".ljust(29) + + "Test".center(9) + "Total".center(9)) print("=" * 80) # make a note of tests start time
Signature · 1 line
2.14.4
PATCH v3 2/9 autotest: fix invalid code on reports
In reply to Anatoly BurakovThere are no reports defined for any test, so this codepath was never triggered, but it's still wrong because it's referencing variables that aren't there. Fix it by passing target into the test function, and reference correct log variable.
--- test/test/autotest_runner.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
test/test/autotest_runner.py +7 -5
@@ -41,7 +41,7 @@ def wait_prompt(child): # quite a bit of effort to make it work). -def run_test_group(cmdline, test_group): +def run_test_group(cmdline, target, test_group): results = [] child = None start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group): # make a note when the test was finished end_time = time.time() + log = logfile.getvalue() + # append test data to the result tuple - result += (test["Name"], end_time - start_time, - logfile.getvalue()) + result += (test["Name"], end_time - start_time, log) # call report function, if any defined, and supply it with # target and complete log for test run if test["Report"]: - report = test["Report"](self.target, log) + report = test["Report"](target, log) # append report to results tuple result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self): for test_group in self.parallel_test_groups: result = pool.apply_async(run_test_group, [self.__get_cmdline(test_group), + self.target, test_group]) results.append(result)
@@ -367,7 +369,7 @@ def run_all_tests(self): # run non_parallel tests. they are run one by one, synchronously for test_group in self.non_parallel_test_groups: group_result = run_test_group( - self.__get_cmdline(test_group), test_group) + self.__get_cmdline(test_group), self.target, test_group) self.__process_results(group_result)
Signature · 1 line
2.14.4
PATCH v3 3/9 autotest: make autotest runner python 2/3 compliant
In reply to Anatoly BurakovAutotest runner was still using python 2-style print syntax. Fix it by importing print function from the future, and fix the calls to be python-3 style.
--- test/test/autotest_runner.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
test/test/autotest_runner.py +4 -3
@@ -3,6 +3,7 @@ # The main logic behind running autotests in parallel +from __future__ import print_function import StringIO import csv import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group): # prepare logging of init startuplog = StringIO.StringIO() - print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"]) - print >>startuplog, "\ncmdline=%s" % cmdline + print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog) + print("\ncmdline=%s" % cmdline, file=startuplog) child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group): try: # print test name to log buffer - print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"]) + print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test if stripped or test["Command"] in avail_cmds:
Signature · 1 line
2.14.4
PATCH v3 4/9 autotest: visually separate different test categories
In reply to Anatoly BurakovHelp visually identify parallel vs. non-parallel autotests.
--- test/test/autotest_runner.py | 2 ++ 1 file changed, 2 insertions(+)
test/test/autotest_runner.py +2 -0
@@ -341,6 +341,7 @@ def run_all_tests(self): # make a note of tests start time self.start = time.time() + print("Parallel autotests:") # assign worker threads to run test groups for test_group in self.parallel_test_groups: result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self): # remove result from results list once we're done with it results.remove(group_result) + print("Non-parallel autotests:") # run non_parallel tests. they are run one by one, synchronously for test_group in self.non_parallel_test_groups: group_result = run_test_group(
Signature · 1 line
2.14.4
PATCH v3 5/9 autotest: improve filtering
In reply to Anatoly BurakovImprove code for filtering test groups. Also, move reading binary symbols into filtering stage, so that tests that are meant to be skipped are never attempted to be executed in the first place. Before running tests, print out any tests that were skipped because they weren't compiled.
--- test/test/autotest_runner.py | 118 ++++++++++++++++++++++++------------------- 1 file changed, 66 insertions(+), 52 deletions(-)
test/test/autotest_runner.py +66 -52
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group): results.append((0, "Success", "Start %s" % test_group["Prefix"], time.time() - start_time, startuplog.getvalue(), None)) - # parse the binary for available test commands - binary = cmdline.split()[0] - stripped = 'not stripped' not in subprocess.check_output(['file', binary]) - if not stripped: - symbols = subprocess.check_output(['nm', binary]).decode('utf-8') - avail_cmds = re.findall('test_register_(\w+)', symbols) - # run all tests in test group for test in test_group["Tests"]:
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group): print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test - if stripped or test["Command"] in avail_cmds: - result = test["Func"](child, test["Command"]) - else: - result = (0, "Skipped [Not Available]") + result = test["Func"](child, test["Command"]) # make a note when the test was finished end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner: def __init__(self, cmdline, target, blacklist, whitelist): self.cmdline = cmdline self.target = target + self.binary = cmdline.split()[0] self.blacklist = blacklist self.whitelist = whitelist + self.skipped = [] # log file filename logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results): if i != 0: self.csvwriter.writerow([test_name, test_result, result_str]) - # this function iterates over test groups and removes each - # test that is not in whitelist/blacklist - def __filter_groups(self, test_groups): - groups_to_remove = [] - - # filter out tests from parallel test groups - for i, test_group in enumerate(test_groups): - - # iterate over a copy so that we could safely delete individual - # tests - for test in test_group["Tests"][:]: - test_id = test["Command"] - - # dump tests are specified in full e.g. "Dump_mempool" - if "_autotest" in test_id: - test_id = test_id[:-len("_autotest")] - - # filter out blacklisted/whitelisted tests - if self.blacklist and test_id in self.blacklist: - test_group["Tests"].remove(test) - continue - if self.whitelist and test_id not in self.whitelist: - test_group["Tests"].remove(test) - continue - - # modify or remove original group - if len(test_group["Tests"]) > 0: - test_groups[i] = test_group - else: - # remember which groups should be deleted - # put the numbers backwards so that we start - # deleting from the end, not from the beginning - groups_to_remove.insert(0, i) + # this function checks individual test and decides if this test should be in + # the group by comparing it against whitelist/blacklist. it also checks if + # the test is compiled into the binary, and marks it as skipped if necessary + def __filter_test(self, test): + test_cmd = test["Command"] + test_id = test_cmd + + # dump tests are specified in full e.g. "Dump_mempool" + if "_autotest" in test_id: + test_id = test_id[:-len("_autotest")] + + # filter out blacklisted/whitelisted tests + if self.blacklist and test_id in self.blacklist: + return False + if self.whitelist and test_id not in self.whitelist: + return False + + # if test wasn't compiled in, remove it as well + + # parse the binary for available test commands + stripped = 'not stripped' not in \ + subprocess.check_output(['file', self.binary]) + if not stripped: + symbols = subprocess.check_output(['nm', + self.binary]).decode('utf-8') + avail_cmds = re.findall('test_register_(\w+)', symbols) + + if test_cmd not in avail_cmds: + # notify user + result = 0, "Skipped [Not compiled]", test_id, 0, "", None + self.skipped.append(tuple(result)) + return False - # remove test groups that need to be removed - for i in groups_to_remove: - del test_groups[i] + return True - return test_groups + def __filter_group(self, group): + group["Tests"] = list(filter(self.__filter_test, group["Tests"])) + return len(group["Tests"]) > 0 # iterate over test groups and run tests associated with them def run_all_tests(self): # filter groups - self.parallel_test_groups = \ - self.__filter_groups(self.parallel_test_groups) - self.non_parallel_test_groups = \ - self.__filter_groups(self.non_parallel_test_groups) + # for each test group, check all tests against the filter, then remove + # all groups that don't have any tests + self.parallel_test_groups = list( + filter(self.__filter_group, + self.parallel_test_groups) + ) + self.non_parallel_test_groups = list( + filter(self.__filter_group, + self.non_parallel_test_groups) + ) # create a pool of worker threads pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self): "Test".center(9) + "Total".center(9)) print("=" * 80) + # print out skipped autotests if there were any + if len(self.skipped): + print("Skipped autotests:") + + # print out any skipped tests + for result in self.skipped: + # unpack result tuple + test_result, result_str, test_name, _, _, _ = result + self.csvwriter.writerow([test_name, test_result, + result_str]) + + t = ("%s:" % test_name).ljust(30) + t += result_str.ljust(29) + t += "[00m 00s]" + + print(t) + # make a note of tests start time self.start = time.time()
Signature · 1 line
2.14.4
PATCH v3 6/9 autotest: remove autotest grouping
In reply to Anatoly BurakovPreviously, all autotests were grouped into (seemingly arbitrary) groups. The goal was to run all tests in parallel (so that autotest finishes faster), but we couldn't just do it willy-nilly because DPDK couldn't allocate and free hugepages on-demand, so we had to find autotest groupings that could work memory-wise and still be fast enough to not hold up shorter tests. The inflexibility of memory subsystem has now been fixed for 18.05, so grouping autotests is no longer necessary.
Thus, this commit moves all autotests into two groups - parallel(izable) autotests, and non-arallel(izable) autotests (typically performance tests). Note that this particular commit makes running autotests dog slow because while the tests are now in a single group, the test function itself hasn't changed much, so all autotests are now run one-by-one, starting and stopping the DPDK test application.
--- test/test/autotest.py | 7 +- test/test/autotest_data.py | 749 +++++++++++++++++-------------------------- test/test/autotest_runner.py | 271 ++++++---------- 3 files changed, 408 insertions(+), 619 deletions(-)
test/test/autotest.py +2 -5
@@ -39,11 +39,8 @@ def usage(): runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist, test_whitelist) -for test_group in autotest_data.parallel_test_group_list: - runner.add_parallel_test_group(test_group) - -for test_group in autotest_data.non_parallel_test_group_list: - runner.add_non_parallel_test_group(test_group) +runner.parallel_tests = autotest_data.parallel_test_list[:] +runner.non_parallel_tests = autotest_data.non_parallel_test_list[:] num_fails = runner.run_all_tests()
test/test/autotest_data.py +303 -446
768 lines not on this page · raw message
test/test/autotest_runner.py +103 -168
356 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH v3 7/9 autotest: properly parallelize unit tests
In reply to Anatoly BurakovNow that everything else is in place, we can run unit tests in a different fashion to what they were running as before. Previously, we had all autotests as part of groups (largely obtained through trial and error) to ensure parallel execution while still limiting amounts of memory used by those tests.
This is no longer necessary, and as of previous commit, all tests are now in the same group (still broken into two categories). They still run one-by-one though. Fix this by initializing child processes in multiprocessing Pool initialization, and putting all tests on the queue, so that tests are executed by the first idle worker. Tests are also affinitized to different NUMA nodes using taskset in a round-robin fashion, to prevent over-exhausting memory on any given NUMA node.
Non-parallel tests are executed in similar fashion, but on a separate queue which will have only one pool worker, ensuring non-parallel execution.
Support for FreeBSD is also added to ensure that on FreeBSD, all tests are run sequentially even for the parallel section.
--- test/test/autotest.py | 6 +- test/test/autotest_runner.py | 277 +++++++++++++++++++++++++++---------------- 2 files changed, 183 insertions(+), 100 deletions(-)
test/test/autotest.py +5 -1
@@ -36,8 +36,12 @@ def usage(): print(cmdline) +# how many workers to run tests with. FreeBSD doesn't support multiple primary +# processes, so make it 1, otherwise make it 4. ignored for non-parallel tests +n_processes = 1 if "bsdapp" in target else 4 + runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist, - test_whitelist) + test_whitelist, n_processes) runner.parallel_tests = autotest_data.parallel_test_list[:] runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
test/test/autotest_runner.py +178 -99
369 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH v3 8/9 autotest: update autotest test case list
In reply to Anatoly BurakovAutotest is enhanced with additional test cases
being added to autotest_data.py
Removed non existing PCI autotest.
--- test/test/autotest_data.py | 368 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 360 insertions(+), 8 deletions(-)
test/test/autotest_data.py +360 -8
@@ -134,12 +134,6 @@ "Func": default_autotest, "Report": None, }, - { - "Name": "PCI autotest", - "Command": "pci_autotest", - "Func": default_autotest, - "Report": None, - }, { "Name": "Malloc autotest", "Command": "malloc_autotest",
@@ -248,6 +242,309 @@ "Func": default_autotest, "Report": None, }, + { + "Name": "Eventdev selftest octeontx", + "Command": "eventdev_selftest_octeontx", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Event ring autotest", + "Command": "event_ring_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Table autotest", + "Command": "table_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Flow classify autotest", + "Command": "flow_classify_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx mode", + "Command": "set_rxtx_mode", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx anchor", + "Command": "set_rxtx_anchor", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx sc", + "Command": "set_rxtx_sc", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Event eth rx adapter autotest", + "Command": "event_eth_rx_adapter_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Rawdev autotest", + "Command": "rawdev_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Kvargs autotest", + "Command": "kvargs_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Devargs autotest", + "Command": "devargs_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding autotest", + "Command": "link_bonding_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding mode4 autotest", + "Command": "link_bonding_mode4_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding rssconf autotest", + "Command": "link_bonding_rssconf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Crc autotest", + "Command": "crc_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Distributor autotest", + "Command": "distributor_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reorder autotest", + "Command": "reorder_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Barrier autotest", + "Command": "barrier_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "User delay", + "Command": "user_delay_us", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Bitmap test", + "Command": "bitmap_test", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash scaling autotest", + "Command": "hash_scaling_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash multiwriter autotest", + "Command": "hash_multiwriter_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Service autotest", + "Command": "service_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Timer racecond autotest", + "Command": "timer_racecond_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Member autotest", + "Command": "member_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Efd_autotest", + "Command": "efd_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Thash autotest", + "Command": "thash_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash function autotest", + "Command": "hash_functions_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev sw mrvl autotest", + "Command": "cryptodev_sw_mrvl_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev dpaa2 sec autotest", + "Command": "cryptodev_dpaa2_sec_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev dpaa sec autotest", + "Command": "cryptodev_dpaa_sec_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev qat autotest", + "Command": "cryptodev_qat_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev aesni mb autotest", + "Command": "cryptodev_aesni_mb_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev openssl autotest", + "Command": "cryptodev_openssl_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev scheduler autotest", + "Command": "cryptodev_scheduler_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev aesni gcm autotest", + "Command": "cryptodev_aesni_gcm_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev null autotest", + "Command": "cryptodev_null_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev sw snow3g autotest", + "Command": "cryptodev_sw_snow3g_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev sw kasumi autotest", + "Command": "cryptodev_sw_kasumi_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev_sw_zuc_autotest", + "Command": "cryptodev_sw_zuc_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reciprocal division", + "Command": "reciprocal_division", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Red all", + "Command": "red_all", + "Func": default_autotest, + "Report": None, + }, + # + #Please always keep all dump tests at the end and together! + # + { + "Name": "Dump physmem", + "Command": "dump_physmem", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump memzone", + "Command": "dump_memzone", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump struct sizes", + "Command": "dump_struct_sizes", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump mempool", + "Command": "dump_mempool", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump malloc stats", + "Command": "dump_malloc_stats", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump devargs", + "Command": "dump_devargs", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump log types", + "Command": "dump_log_types", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump_ring", + "Command": "dump_ring", + "Func": dump_autotest, + "Report": None, + }, ] # tests that should not be run when any other tests are running
@@ -259,8 +556,8 @@ "Report": None, }, { - "Name": "Eventdev sw autotest", - "Command": "eventdev_sw_autotest", + "Name": "Eventdev selftest sw", + "Command": "eventdev_selftest_sw", "Func": default_autotest, "Report": None, },
@@ -312,6 +609,61 @@ "Func": default_autotest, "Report": None, }, + { + + "Name": "Pmd perf autotest", + "Command": "pmd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Ring pmd perf autotest", + "Command": "ring_pmd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Distributor perf autotest", + "Command": "distributor_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Red_perf", + "Command": "red_perf", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Lpm6 perf autotest", + "Command": "lpm6_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Lpm perf autotest", + "Command": "lpm_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Efd perf autotest", + "Command": "efd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Member perf autotest", + "Command": "member_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reciprocal division perf", + "Command": "reciprocal_division_perf", + "Func": default_autotest, + "Report": None, + }, # # Please always make sure that ring_perf is the last test! #
Signature · 1 line
2.14.4
Re: [PATCH v3 8/9] autotest: update autotest test case list
Quoted 10 lines · Reshma Pattan
Autotest is enhanced with additional test cases
being added toautotest_data.pyRemoved non existing PCI autotest.
Signed-off-by Reshma Pattan <[email protected]>Signed-off-by Jananee Parthasarathy <[email protected]>---
<snip>
Quoted 15 lines
test/test/autotest_data.py +15
-0+ }, + { + "Name": "Flow classify autotest", + "Command": "flow_classify_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx mode", + "Command": "set_rxtx_mode", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx anchor",
test/test/autotest_data.py line 286
Quoted 10 lines
test/test/autotest_data.py +10
-0+ "Command": "set_rxtx_anchor", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx sc", + "Command": "set_rxtx_sc", + "Func": default_autotest, + "Report": None, + },
The above three tests don't look like autotests to me. I have no idea what they are for, but either they need a special function, or they need to be taken out.
Quoted 12 lines
test/test/autotest_data.py +12
-0+ { + "Name": "Event eth rx adapter autotest", + "Command": "event_eth_rx_adapter_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Rawdev autotest", + "Command": "rawdev_autotest", + "Func": default_autotest, + "Report": None, + },
<snip>
Quoted 4 lines
test/test/autotest_data.py +4
-0+ "Report": None, + }, + { + "Name": "Barrier autotest",
test/test/autotest_data.py line 358
Quoted 10 lines
test/test/autotest_data.py +10
-0+ "Command": "barrier_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "User delay", + "Command": "user_delay_us", + "Func": default_autotest, + "Report": None, + },
This doesn't look like autotests to me. I have no idea what it is for, but either it needs a special function, or it needs to be taken out.
Quoted 15 lines
test/test/autotest_data.py +15
-0+ { + "Name": "Bitmap test", + "Command": "bitmap_test", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash scaling autotest", + "Command": "hash_scaling_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash multiwriter autotest", + "Command": "hash_multiwriter_autotest",
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v3 8/9] autotest: update autotest test case list
Hi,
Quoted 29 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, July 16, 2018 4:16 PM
To: Pattan, Reshma <[email protected]>; [email protected]; [email protected]
Cc: Parthasarathy, JananeeX M <[email protected]>; [email protected]
Subject: Re: [PATCH v3 8/9] autotest: update autotest test case listQuoted 8 lines
test/test/autotest_data.py +8
-0+ { + "Name": "Set rxtx mode", + "Command": "set_rxtx_mode", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx anchor",test/test/autotest_data.py line 286
Quoted 10 lines
test/test/autotest_data.py +10
-0+ "Command": "set_rxtx_anchor", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx sc", + "Command": "set_rxtx_sc", + "Func": default_autotest, + "Report": None, + },
Quoted 4 lines
The above three tests don't look like autotests to me. I have no idea what they are for, but either they need a special function, or they need to be taken out.
These commands needs to be run manually from test cmd prompt to various set rxtx mode, rxtx rate and rxtx direction . These can be used to verify pmd perf test with vaiours set of above values.
So this can be removed from autotest.
Quoted 9 lines
test/test/autotest_data.py line 358
Quoted 5 lines
test/test/autotest_data.py +5
-0+ "Name": "User delay", + "Command": "user_delay_us", + "Func": default_autotest, + "Report": None, + },This doesn't look like autotests to me. I have no idea what it is for, but either it needs a special function, or it needs to be taken out.
This is autotest but the name does'nt have the autotest name in it. So I will retain this.
Thanks,
Reshma
Re: [PATCH v3 8/9] autotest: update autotest test case list
Quoted 36 lines · Reshma
Hi,
Quoted 29 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, July 16, 2018 4:16 PM
To: Pattan, Reshma <[email protected]>; [email protected]; [email protected]
Cc: Parthasarathy, JananeeX M <[email protected]>; [email protected]
Subject: Re: [PATCH v3 8/9] autotest: update autotest test case listQuoted 8 lines
test/test/autotest_data.py +8
-0+ { + "Name": "Set rxtx mode", + "Command": "set_rxtx_mode", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx anchor",test/test/autotest_data.py line 286
Quoted 10 lines
test/test/autotest_data.py +10
-0+ "Command": "set_rxtx_anchor", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx sc", + "Command": "set_rxtx_sc", + "Func": default_autotest, + "Report": None, + },Quoted 4 lines
The above three tests don't look like autotests to me. I have no idea what they are for, but either they need a special function, or they need to be taken out.
Quoted 4 lines
These commands needs to be run manually from test cmd prompt to various set rxtx mode, rxtx rate and rxtx direction . These can be used to verify pmd perf test with vaiours set of above values.
So this can be removed from autotest.
We do have PMD perf tests in the script - do they call these functions? If they are required for PMD autotests, maybe PMD autotests deserve a special test function calling these commands before running the tests?
(if they also work without these commands, then we can perhaps postpone this to 18.11)
Quoted 11 lines
Quoted 10 lines
test/test/autotest_data.py line 358
Quoted 6 lines
test/test/autotest_data.py +5
-0+ "Name": "User delay", + "Command": "user_delay_us", + "Func": default_autotest, + "Report": None, + },This doesn't look like autotests to me. I have no idea what it is for, but either it needs a special function, or it needs to be taken out.
This is autotest but the name does'nt have the autotest name in it. So I will retain this.
OK.
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v3 8/9] autotest: update autotest test case list
Hi,
Quoted 50 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Tuesday, July 17, 2018 10:23 AM
To: Pattan, Reshma <[email protected]>; [email protected]; [email protected]
Cc: Parthasarathy, JananeeX M <[email protected]>; [email protected]
Subject: Re: [PATCH v3 8/9] autotest: update autotest test case listQuoted 31 lines · Reshma
Hi,
Quoted 29 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, July 16, 2018 4:16 PM
To: Pattan, Reshma <[email protected]>; [email protected]; [email protected]
Cc: Parthasarathy, JananeeX M <[email protected]>; [email protected]
Subject: Re: [PATCH v3 8/9] autotest: update autotest test case listQuoted 18 lines
test/test/autotest_data.py +18
-0+ { + "Name": "Set rxtx mode", + "Command": "set_rxtx_mode", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx anchor", + "Command": "set_rxtx_anchor", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx sc", + "Command": "set_rxtx_sc", + "Func": default_autotest, + "Report": None, + },Quoted 6 lines
Quoted 4 lines
The above three tests don't look like autotests to me. I have no idea what they are for, but either they need a special function, or they need to be taken out.
These commands needs to be run manually from test cmd prompt to
various set rxtx mode, rxtx rate and rxtx direction .
Quoted 2 lines
These can be used to verify pmd perf test with vaiours set of above values.
Quoted 9 lines
Quoted 1 line
So this can be removed from autotest.
We do have PMD perf tests in the script - do they call these functions? If they are required for PMD autotests, maybe PMD autotests deserve a special test function calling these commands before running the tests?
(if they also work without these commands, then we can perhaps postpone this to 18.11)
I ran pmd perf test manually and it passes without having to use above set_rxtx commands.
Thanks,
Reshma
Burakov, Anatoly
Reviewed-byRe: [PATCH v3 8/9] autotest: update autotest test case list
Quoted 58 lines · Reshma
Hi,
Quoted 56 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Tuesday, July 17, 2018 10:23 AM
To: Pattan, Reshma <[email protected]>; [email protected]; [email protected]
Cc: Parthasarathy, JananeeX M <[email protected]>; [email protected]
Subject: Re: [PATCH v3 8/9] autotest: update autotest test case listQuoted 31 lines · Reshma
Hi,
Quoted 29 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, July 16, 2018 4:16 PM
To: Pattan, Reshma <[email protected]>; [email protected]; [email protected]
Cc: Parthasarathy, JananeeX M <[email protected]>; [email protected]
Subject: Re: [PATCH v3 8/9] autotest: update autotest test case listQuoted 18 lines
test/test/autotest_data.py +18
-0+ { + "Name": "Set rxtx mode", + "Command": "set_rxtx_mode", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx anchor", + "Command": "set_rxtx_anchor", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Set rxtx sc", + "Command": "set_rxtx_sc", + "Func": default_autotest, + "Report": None, + },Quoted 6 lines
Quoted 4 lines
The above three tests don't look like autotests to me. I have no idea what they are for, but either they need a special function, or they need to be taken out.
These commands needs to be run manually from test cmd prompt to
various set rxtx mode, rxtx rate and rxtx direction .
Quoted 3 lines
These can be used to verify pmd perf test with vaiours set of above values.
So this can be removed from autotest.
We do have PMD perf tests in the script - do they call these functions? If they are required for PMD autotests, maybe PMD autotests deserve a special test function calling these commands before running the tests?
Quoted 9 lines
Quoted 3 lines
(if they also work without these commands, then we can perhaps postpone this to 18.11)
I ran pmd perf test manually and it passes without having to use above
set_rxtxcommands.Thanks,
Reshma
Great.
Signature · 2 lines
Thanks, Anatoly
v3 cover Make unit tests great again
In reply to Anatoly BurakovPreviously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.
In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.
This patchset merges changes done in [1], [2]
[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://patches.dpdk.org/patch/40373/
v3: Moved all dump tests together to the end of the parallel list.
Updated the commit message about PCI auto test removal. Removed the unnecessary changes of the patch
http://patches.dpdk.org/patch/40371/
Reshma Pattan (9):
autotest: fix printing
autotest: fix invalid code on reports
autotest: make autotest runner python 2/3 compliant
autotest: visually separate different test categories
autotest: improve filtering
autotest: remove autotest grouping
autotest: properly parallelize unit tests
autotest: update autotest test case list
mk: update make targets for classified testcases
mk/rte.sdkroot.mk | 4 +- mk/rte.sdktest.mk | 33 +- test/test/autotest.py | 13 +- test/test/autotest_data.py | 1099 +++++++++++++++++++++++++----------------- test/test/autotest_runner.py | 519 ++++++++++---------- 5 files changed, 966 insertions(+), 702 deletions(-)
Signature · 1 line
2.14.4
PATCH v3 9/9 mk: update make targets for classified testcases
In reply to Anatoly BurakovMakefiles are updated with new test case lists.
Test cases are classified as -
P1 - Main test cases,
P2 - Cryptodev/driver test cases,
P3 - Perf test cases which takes longer than 10s,
P4 - Logging/Dump test cases.
Makefile is updated with different targets
for the above classified groups.
Test cases for different targets are listed accordingly.
--- mk/rte.sdkroot.mk | 4 ++-- mk/rte.sdktest.mk | 33 +++++++++++++++++++++++++++------ 2 files changed, 29 insertions(+), 8 deletions(-)
mk/rte.sdkroot.mk +2 -2
@@ -68,8 +68,8 @@ config defconfig showconfigs showversion showversionum: cscope gtags tags etags: $(Q)$(RTE_SDK)/devtools/build-tags.sh $@ $T -.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage -test test-basic test-fast test-ring test-mempool test-perf coverage: +.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump +test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump: $(Q)$(MAKE) -f $(RTE_SDK)/mk/rte.sdktest.mk $@ test: test-build
mk/rte.sdktest.mk +27 -6
@@ -18,14 +18,35 @@ DIR := $(shell basename $(RTE_OUTPUT)) # # test: launch auto-tests, very simple for now. # -.PHONY: test test-basic test-fast test-perf coverage +.PHONY: test test-basic test-fast test-perf test-drivers test-dump coverage -PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf -coverage: BLACKLIST=-$(PERFLIST) -test-fast: BLACKLIST=-$(PERFLIST) -test-perf: WHITELIST=$(PERFLIST) +PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf,\ + reciprocal_division,reciprocal_division_perf,lpm_perf,red_all,\ + barrier,hash_multiwriter,timer_racecond,efd,hash_functions,\ + eventdev_selftest_sw,member_perf,efd_perf,lpm6_perf,red_perf,\ + distributor_perf,ring_pmd_perf,pmd_perf,ring_perf +DRIVERSLIST=link_bonding,link_bonding_mode4,link_bonding_rssconf,\ + cryptodev_sw_mrvl,cryptodev_dpaa2_sec,cryptodev_dpaa_sec,\ + cryptodev_qat,cryptodev_aesni_mb,cryptodev_openssl,\ + cryptodev_scheduler,cryptodev_aesni_gcm,cryptodev_null,\ + cryptodev_sw_snow3g,cryptodev_sw_kasumi,cryptodev_sw_zuc +DUMPLIST=dump_struct_sizes,dump_mempool,dump_malloc_stats,dump_devargs,\ + dump_log_types,dump_ring,quit,dump_physmem,dump_memzone,\ + devargs_autotest -test test-basic test-fast test-perf: +SPACESTR:= +SPACESTR+= +STRIPPED_PERFLIST=$(subst $(SPACESTR),,$(PERFLIST)) +STRIPPED_DRIVERSLIST=$(subst $(SPACESTR),,$(DRIVERSLIST)) +STRIPPED_DUMPLIST=$(subst $(SPACESTR),,$(DUMPLIST)) + +coverage: BLACKLIST=-$(STRIPPED_PERFLIST) +test-fast: BLACKLIST=-$(STRIPPED_PERFLIST),$(STRIPPED_DRIVERSLIST),$(STRIPPED_DUMPLIST) +test-perf: WHITELIST=$(STRIPPED_PERFLIST) +test-drivers: WHITELIST=$(STRIPPED_DRIVERSLIST) +test-dump: WHITELIST=$(STRIPPED_DUMPLIST) + +test test-basic test-fast test-perf test-drivers test-dump: @mkdir -p $(AUTOTEST_DIR) ; \ cd $(AUTOTEST_DIR) ; \ if [ -f $(RTE_OUTPUT)/app/test ]; then \
Signature · 1 line
2.14.4
PATCH v4 1/9 autotest: fix printing
In reply to Anatoly BurakovPreviously, printing was done using tuple syntax, which caused output to appear as a tuple as opposed to being one string. Fix this by using addition operator instead.
--- test/test/autotest_runner.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
test/test/autotest_runner.py +3 -3
@@ -247,7 +247,7 @@ def __process_results(self, results): # don't print out total time every line, it's the same anyway if i == len(results) - 1: - print(result, + print(result + "[%02dm %02ds]" % (total_time / 60, total_time % 60)) else: print(result)
@@ -332,8 +332,8 @@ def run_all_tests(self): # create table header print("") - print("Test name".ljust(30), "Test result".ljust(29), - "Test".center(9), "Total".center(9)) + print("Test name".ljust(30) + "Test result".ljust(29) + + "Test".center(9) + "Total".center(9)) print("=" * 80) # make a note of tests start time
Signature · 1 line
2.14.4
PATCH v4 2/9 autotest: fix invalid code on reports
In reply to Anatoly BurakovThere are no reports defined for any test, so this codepath was never triggered, but it's still wrong because it's referencing variables that aren't there. Fix it by passing target into the test function, and reference correct log variable.
--- test/test/autotest_runner.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
test/test/autotest_runner.py +7 -5
@@ -41,7 +41,7 @@ def wait_prompt(child): # quite a bit of effort to make it work). -def run_test_group(cmdline, test_group): +def run_test_group(cmdline, target, test_group): results = [] child = None start_time = time.time()
@@ -128,14 +128,15 @@ def run_test_group(cmdline, test_group): # make a note when the test was finished end_time = time.time() + log = logfile.getvalue() + # append test data to the result tuple - result += (test["Name"], end_time - start_time, - logfile.getvalue()) + result += (test["Name"], end_time - start_time, log) # call report function, if any defined, and supply it with # target and complete log for test run if test["Report"]: - report = test["Report"](self.target, log) + report = test["Report"](target, log) # append report to results tuple result += (report,)
@@ -343,6 +344,7 @@ def run_all_tests(self): for test_group in self.parallel_test_groups: result = pool.apply_async(run_test_group, [self.__get_cmdline(test_group), + self.target, test_group]) results.append(result)
@@ -367,7 +369,7 @@ def run_all_tests(self): # run non_parallel tests. they are run one by one, synchronously for test_group in self.non_parallel_test_groups: group_result = run_test_group( - self.__get_cmdline(test_group), test_group) + self.__get_cmdline(test_group), self.target, test_group) self.__process_results(group_result)
Signature · 1 line
2.14.4
PATCH v4 3/9 autotest: make autotest runner python 2/3 compliant
In reply to Anatoly BurakovAutotest runner was still using python 2-style print syntax. Fix it by importing print function from the future, and fix the calls to be python-3 style.
--- test/test/autotest_runner.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
test/test/autotest_runner.py +4 -3
@@ -3,6 +3,7 @@ # The main logic behind running autotests in parallel +from __future__ import print_function import StringIO import csv import multiprocessing
@@ -52,8 +53,8 @@ def run_test_group(cmdline, target, test_group): # prepare logging of init startuplog = StringIO.StringIO() - print >>startuplog, "\n%s %s\n" % ("=" * 20, test_group["Prefix"]) - print >>startuplog, "\ncmdline=%s" % cmdline + print("\n%s %s\n" % ("=" * 20, test_group["Prefix"]), file=startuplog) + print("\ncmdline=%s" % cmdline, file=startuplog) child = pexpect.spawn(cmdline, logfile=startuplog)
@@ -117,7 +118,7 @@ def run_test_group(cmdline, target, test_group): try: # print test name to log buffer - print >>logfile, "\n%s %s\n" % ("-" * 20, test["Name"]) + print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test if stripped or test["Command"] in avail_cmds:
Signature · 1 line
2.14.4
PATCH v4 4/9 autotest: visually separate different test categories
In reply to Anatoly BurakovHelp visually identify parallel vs. non-parallel autotests.
--- test/test/autotest_runner.py | 2 ++ 1 file changed, 2 insertions(+)
test/test/autotest_runner.py +2 -0
@@ -341,6 +341,7 @@ def run_all_tests(self): # make a note of tests start time self.start = time.time() + print("Parallel autotests:") # assign worker threads to run test groups for test_group in self.parallel_test_groups: result = pool.apply_async(run_test_group,
@@ -367,6 +368,7 @@ def run_all_tests(self): # remove result from results list once we're done with it results.remove(group_result) + print("Non-parallel autotests:") # run non_parallel tests. they are run one by one, synchronously for test_group in self.non_parallel_test_groups: group_result = run_test_group(
Signature · 1 line
2.14.4
PATCH v4 5/9 autotest: improve filtering
In reply to Anatoly BurakovImprove code for filtering test groups. Also, move reading binary symbols into filtering stage, so that tests that are meant to be skipped are never attempted to be executed in the first place. Before running tests, print out any tests that were skipped because they weren't compiled.
--- test/test/autotest_runner.py | 118 ++++++++++++++++++++++++------------------- 1 file changed, 66 insertions(+), 52 deletions(-)
test/test/autotest_runner.py +66 -52
@@ -95,13 +95,6 @@ def run_test_group(cmdline, target, test_group): results.append((0, "Success", "Start %s" % test_group["Prefix"], time.time() - start_time, startuplog.getvalue(), None)) - # parse the binary for available test commands - binary = cmdline.split()[0] - stripped = 'not stripped' not in subprocess.check_output(['file', binary]) - if not stripped: - symbols = subprocess.check_output(['nm', binary]).decode('utf-8') - avail_cmds = re.findall('test_register_(\w+)', symbols) - # run all tests in test group for test in test_group["Tests"]:
@@ -121,10 +114,7 @@ def run_test_group(cmdline, target, test_group): print("\n%s %s\n" % ("-" * 20, test["Name"]), file=logfile) # run test function associated with the test - if stripped or test["Command"] in avail_cmds: - result = test["Func"](child, test["Command"]) - else: - result = (0, "Skipped [Not Available]") + result = test["Func"](child, test["Command"]) # make a note when the test was finished end_time = time.time()
@@ -186,8 +176,10 @@ class AutotestRunner: def __init__(self, cmdline, target, blacklist, whitelist): self.cmdline = cmdline self.target = target + self.binary = cmdline.split()[0] self.blacklist = blacklist self.whitelist = whitelist + self.skipped = [] # log file filename logfile = "%s.log" % target
@@ -276,53 +268,58 @@ def __process_results(self, results): if i != 0: self.csvwriter.writerow([test_name, test_result, result_str]) - # this function iterates over test groups and removes each - # test that is not in whitelist/blacklist - def __filter_groups(self, test_groups): - groups_to_remove = [] - - # filter out tests from parallel test groups - for i, test_group in enumerate(test_groups): - - # iterate over a copy so that we could safely delete individual - # tests - for test in test_group["Tests"][:]: - test_id = test["Command"] - - # dump tests are specified in full e.g. "Dump_mempool" - if "_autotest" in test_id: - test_id = test_id[:-len("_autotest")] - - # filter out blacklisted/whitelisted tests - if self.blacklist and test_id in self.blacklist: - test_group["Tests"].remove(test) - continue - if self.whitelist and test_id not in self.whitelist: - test_group["Tests"].remove(test) - continue - - # modify or remove original group - if len(test_group["Tests"]) > 0: - test_groups[i] = test_group - else: - # remember which groups should be deleted - # put the numbers backwards so that we start - # deleting from the end, not from the beginning - groups_to_remove.insert(0, i) + # this function checks individual test and decides if this test should be in + # the group by comparing it against whitelist/blacklist. it also checks if + # the test is compiled into the binary, and marks it as skipped if necessary + def __filter_test(self, test): + test_cmd = test["Command"] + test_id = test_cmd + + # dump tests are specified in full e.g. "Dump_mempool" + if "_autotest" in test_id: + test_id = test_id[:-len("_autotest")] + + # filter out blacklisted/whitelisted tests + if self.blacklist and test_id in self.blacklist: + return False + if self.whitelist and test_id not in self.whitelist: + return False + + # if test wasn't compiled in, remove it as well + + # parse the binary for available test commands + stripped = 'not stripped' not in \ + subprocess.check_output(['file', self.binary]) + if not stripped: + symbols = subprocess.check_output(['nm', + self.binary]).decode('utf-8') + avail_cmds = re.findall('test_register_(\w+)', symbols) + + if test_cmd not in avail_cmds: + # notify user + result = 0, "Skipped [Not compiled]", test_id, 0, "", None + self.skipped.append(tuple(result)) + return False - # remove test groups that need to be removed - for i in groups_to_remove: - del test_groups[i] + return True - return test_groups + def __filter_group(self, group): + group["Tests"] = list(filter(self.__filter_test, group["Tests"])) + return len(group["Tests"]) > 0 # iterate over test groups and run tests associated with them def run_all_tests(self): # filter groups - self.parallel_test_groups = \ - self.__filter_groups(self.parallel_test_groups) - self.non_parallel_test_groups = \ - self.__filter_groups(self.non_parallel_test_groups) + # for each test group, check all tests against the filter, then remove + # all groups that don't have any tests + self.parallel_test_groups = list( + filter(self.__filter_group, + self.parallel_test_groups) + ) + self.non_parallel_test_groups = list( + filter(self.__filter_group, + self.non_parallel_test_groups) + ) # create a pool of worker threads pool = multiprocessing.Pool(processes=1)
@@ -338,6 +335,23 @@ def run_all_tests(self): "Test".center(9) + "Total".center(9)) print("=" * 80) + # print out skipped autotests if there were any + if len(self.skipped): + print("Skipped autotests:") + + # print out any skipped tests + for result in self.skipped: + # unpack result tuple + test_result, result_str, test_name, _, _, _ = result + self.csvwriter.writerow([test_name, test_result, + result_str]) + + t = ("%s:" % test_name).ljust(30) + t += result_str.ljust(29) + t += "[00m 00s]" + + print(t) + # make a note of tests start time self.start = time.time()
Signature · 1 line
2.14.4
PATCH v4 6/9 autotest: remove autotest grouping
In reply to Anatoly BurakovPreviously, all autotests were grouped into (seemingly arbitrary) groups. The goal was to run all tests in parallel (so that autotest finishes faster), but we couldn't just do it willy-nilly because DPDK couldn't allocate and free hugepages on-demand, so we had to find autotest groupings that could work memory-wise and still be fast enough to not hold up shorter tests. The inflexibility of memory subsystem has now been fixed for 18.05, so grouping autotests is no longer necessary.
Thus, this commit moves all autotests into two groups - parallel(izable) autotests, and non-arallel(izable) autotests (typically performance tests). Note that this particular commit makes running autotests dog slow because while the tests are now in a single group, the test function itself hasn't changed much, so all autotests are now run one-by-one, starting and stopping the DPDK test application.
--- test/test/autotest.py | 7 +- test/test/autotest_data.py | 749 +++++++++++++++++-------------------------- test/test/autotest_runner.py | 271 ++++++---------- 3 files changed, 408 insertions(+), 619 deletions(-)
test/test/autotest.py +2 -5
@@ -39,11 +39,8 @@ def usage(): runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist, test_whitelist) -for test_group in autotest_data.parallel_test_group_list: - runner.add_parallel_test_group(test_group) - -for test_group in autotest_data.non_parallel_test_group_list: - runner.add_non_parallel_test_group(test_group) +runner.parallel_tests = autotest_data.parallel_test_list[:] +runner.non_parallel_tests = autotest_data.non_parallel_test_list[:] num_fails = runner.run_all_tests()
test/test/autotest_data.py +303 -446
768 lines not on this page · raw message
test/test/autotest_runner.py +103 -168
356 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH v4 7/9 autotest: properly parallelize unit tests
In reply to Anatoly BurakovNow that everything else is in place, we can run unit tests in a different fashion to what they were running as before. Previously, we had all autotests as part of groups (largely obtained through trial and error) to ensure parallel execution while still limiting amounts of memory used by those tests.
This is no longer necessary, and as of previous commit, all tests are now in the same group (still broken into two categories). They still run one-by-one though. Fix this by initializing child processes in multiprocessing Pool initialization, and putting all tests on the queue, so that tests are executed by the first idle worker. Tests are also affinitized to different NUMA nodes using taskset in a round-robin fashion, to prevent over-exhausting memory on any given NUMA node.
Non-parallel tests are executed in similar fashion, but on a separate queue which will have only one pool worker, ensuring non-parallel execution.
Support for FreeBSD is also added to ensure that on FreeBSD, all tests are run sequentially even for the parallel section.
--- test/test/autotest.py | 6 +- test/test/autotest_runner.py | 277 +++++++++++++++++++++++++++---------------- 2 files changed, 183 insertions(+), 100 deletions(-)
test/test/autotest.py +5 -1
@@ -36,8 +36,12 @@ def usage(): print(cmdline) +# how many workers to run tests with. FreeBSD doesn't support multiple primary +# processes, so make it 1, otherwise make it 4. ignored for non-parallel tests +n_processes = 1 if "bsdapp" in target else 4 + runner = autotest_runner.AutotestRunner(cmdline, target, test_blacklist, - test_whitelist) + test_whitelist, n_processes) runner.parallel_tests = autotest_data.parallel_test_list[:] runner.non_parallel_tests = autotest_data.non_parallel_test_list[:]
test/test/autotest_runner.py +178 -99
369 lines not on this page · raw message
Signature · 1 line
2.14.4
PATCH v4 8/9 autotest: update autotest test case list
In reply to Anatoly BurakovAutotest is enhanced with additional test cases
being added to autotest_data.py
Removed non existing PCI autotest.
--- test/test/autotest_data.py | 350 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 342 insertions(+), 8 deletions(-)
test/test/autotest_data.py +342 -8
@@ -134,12 +134,6 @@ "Func": default_autotest, "Report": None, }, - { - "Name": "PCI autotest", - "Command": "pci_autotest", - "Func": default_autotest, - "Report": None, - }, { "Name": "Malloc autotest", "Command": "malloc_autotest",
@@ -248,6 +242,291 @@ "Func": default_autotest, "Report": None, }, + { + "Name": "Eventdev selftest octeontx", + "Command": "eventdev_selftest_octeontx", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Event ring autotest", + "Command": "event_ring_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Table autotest", + "Command": "table_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Flow classify autotest", + "Command": "flow_classify_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Event eth rx adapter autotest", + "Command": "event_eth_rx_adapter_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "User delay", + "Command": "user_delay_us", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Rawdev autotest", + "Command": "rawdev_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Kvargs autotest", + "Command": "kvargs_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Devargs autotest", + "Command": "devargs_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding autotest", + "Command": "link_bonding_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding mode4 autotest", + "Command": "link_bonding_mode4_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Link bonding rssconf autotest", + "Command": "link_bonding_rssconf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Crc autotest", + "Command": "crc_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Distributor autotest", + "Command": "distributor_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reorder autotest", + "Command": "reorder_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Barrier autotest", + "Command": "barrier_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Bitmap test", + "Command": "bitmap_test", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash scaling autotest", + "Command": "hash_scaling_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash multiwriter autotest", + "Command": "hash_multiwriter_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Service autotest", + "Command": "service_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Timer racecond autotest", + "Command": "timer_racecond_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Member autotest", + "Command": "member_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Efd_autotest", + "Command": "efd_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Thash autotest", + "Command": "thash_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Hash function autotest", + "Command": "hash_functions_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev sw mrvl autotest", + "Command": "cryptodev_sw_mrvl_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev dpaa2 sec autotest", + "Command": "cryptodev_dpaa2_sec_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev dpaa sec autotest", + "Command": "cryptodev_dpaa_sec_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev qat autotest", + "Command": "cryptodev_qat_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev aesni mb autotest", + "Command": "cryptodev_aesni_mb_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev openssl autotest", + "Command": "cryptodev_openssl_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev scheduler autotest", + "Command": "cryptodev_scheduler_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev aesni gcm autotest", + "Command": "cryptodev_aesni_gcm_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev null autotest", + "Command": "cryptodev_null_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev sw snow3g autotest", + "Command": "cryptodev_sw_snow3g_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev sw kasumi autotest", + "Command": "cryptodev_sw_kasumi_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Cryptodev_sw_zuc_autotest", + "Command": "cryptodev_sw_zuc_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reciprocal division", + "Command": "reciprocal_division", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Red all", + "Command": "red_all", + "Func": default_autotest, + "Report": None, + }, + # + #Please always keep all dump tests at the end and together! + # + { + "Name": "Dump physmem", + "Command": "dump_physmem", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump memzone", + "Command": "dump_memzone", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump struct sizes", + "Command": "dump_struct_sizes", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump mempool", + "Command": "dump_mempool", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump malloc stats", + "Command": "dump_malloc_stats", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump devargs", + "Command": "dump_devargs", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump log types", + "Command": "dump_log_types", + "Func": dump_autotest, + "Report": None, + }, + { + "Name": "Dump_ring", + "Command": "dump_ring", + "Func": dump_autotest, + "Report": None, + }, ] # tests that should not be run when any other tests are running
@@ -259,8 +538,8 @@ "Report": None, }, { - "Name": "Eventdev sw autotest", - "Command": "eventdev_sw_autotest", + "Name": "Eventdev selftest sw", + "Command": "eventdev_selftest_sw", "Func": default_autotest, "Report": None, },
@@ -312,6 +591,61 @@ "Func": default_autotest, "Report": None, }, + { + + "Name": "Pmd perf autotest", + "Command": "pmd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Ring pmd perf autotest", + "Command": "ring_pmd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Distributor perf autotest", + "Command": "distributor_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Red_perf", + "Command": "red_perf", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Lpm6 perf autotest", + "Command": "lpm6_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Lpm perf autotest", + "Command": "lpm_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Efd perf autotest", + "Command": "efd_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Member perf autotest", + "Command": "member_perf_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "Reciprocal division perf", + "Command": "reciprocal_division_perf", + "Func": default_autotest, + "Report": None, + }, # # Please always make sure that ring_perf is the last test! #
Signature · 1 line
2.14.4
v4 cover Make unit tests great again
In reply to Anatoly BurakovPreviously, unit tests were running in groups. There were technical reasons why that was the case (mostly having to do with limiting memory), but it was hard to maintain and update the autotest script.
In 18.05, limiting of memory at DPDK startup was no longer necessary, as DPDK allocates memory at runtime as needed. This has the implication that the old test grouping can now be retired and replaced with a more sensible way of running unit tests (using multiprocessing pool of workers and a queue of tests). This patchset accomplishes exactly that.
This patchset merges changes done in [1], [2]
[1] http://dpdk.org/dev/patchwork/patch/40370/
[2] http://patches.dpdk.org/patch/40373/
v4: Removed non auto tests set_rxtx_mode, set_rxtx_anchor and set_rxtx_sc from autotest_data.py
Reshma Pattan (9):
autotest: fix printing
autotest: fix invalid code on reports
autotest: make autotest runner python 2/3 compliant
autotest: visually separate different test categories
autotest: improve filtering
autotest: remove autotest grouping
autotest: properly parallelize unit tests
autotest: update autotest test case list
mk: update make targets for classified testcases
mk/rte.sdkroot.mk | 4 +- mk/rte.sdktest.mk | 33 +- test/test/autotest.py | 13 +- test/test/autotest_data.py | 1081 +++++++++++++++++++++++++----------------- test/test/autotest_runner.py | 519 ++++++++++---------- 5 files changed, 948 insertions(+), 702 deletions(-)
Signature · 1 line
2.14.4
PATCH v4 9/9 mk: update make targets for classified testcases
In reply to Anatoly BurakovMakefiles are updated with new test case lists.
Test cases are classified as -
P1 - Main test cases,
P2 - Cryptodev/driver test cases,
P3 - Perf test cases which takes longer than 10s,
P4 - Logging/Dump test cases.
Makefile is updated with different targets
for the above classified groups.
Test cases for different targets are listed accordingly.
--- mk/rte.sdkroot.mk | 4 ++-- mk/rte.sdktest.mk | 33 +++++++++++++++++++++++++++------ 2 files changed, 29 insertions(+), 8 deletions(-)
mk/rte.sdkroot.mk +2 -2
@@ -68,8 +68,8 @@ config defconfig showconfigs showversion showversionum: cscope gtags tags etags: $(Q)$(RTE_SDK)/devtools/build-tags.sh $@ $T -.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage -test test-basic test-fast test-ring test-mempool test-perf coverage: +.PHONY: test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump +test test-basic test-fast test-ring test-mempool test-perf coverage test-drivers test-dump: $(Q)$(MAKE) -f $(RTE_SDK)/mk/rte.sdktest.mk $@ test: test-build
mk/rte.sdktest.mk +27 -6
@@ -18,14 +18,35 @@ DIR := $(shell basename $(RTE_OUTPUT)) # # test: launch auto-tests, very simple for now. # -.PHONY: test test-basic test-fast test-perf coverage +.PHONY: test test-basic test-fast test-perf test-drivers test-dump coverage -PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf -coverage: BLACKLIST=-$(PERFLIST) -test-fast: BLACKLIST=-$(PERFLIST) -test-perf: WHITELIST=$(PERFLIST) +PERFLIST=ring_perf,mempool_perf,memcpy_perf,hash_perf,timer_perf,\ + reciprocal_division,reciprocal_division_perf,lpm_perf,red_all,\ + barrier,hash_multiwriter,timer_racecond,efd,hash_functions,\ + eventdev_selftest_sw,member_perf,efd_perf,lpm6_perf,red_perf,\ + distributor_perf,ring_pmd_perf,pmd_perf,ring_perf +DRIVERSLIST=link_bonding,link_bonding_mode4,link_bonding_rssconf,\ + cryptodev_sw_mrvl,cryptodev_dpaa2_sec,cryptodev_dpaa_sec,\ + cryptodev_qat,cryptodev_aesni_mb,cryptodev_openssl,\ + cryptodev_scheduler,cryptodev_aesni_gcm,cryptodev_null,\ + cryptodev_sw_snow3g,cryptodev_sw_kasumi,cryptodev_sw_zuc +DUMPLIST=dump_struct_sizes,dump_mempool,dump_malloc_stats,dump_devargs,\ + dump_log_types,dump_ring,quit,dump_physmem,dump_memzone,\ + devargs_autotest -test test-basic test-fast test-perf: +SPACESTR:= +SPACESTR+= +STRIPPED_PERFLIST=$(subst $(SPACESTR),,$(PERFLIST)) +STRIPPED_DRIVERSLIST=$(subst $(SPACESTR),,$(DRIVERSLIST)) +STRIPPED_DUMPLIST=$(subst $(SPACESTR),,$(DUMPLIST)) + +coverage: BLACKLIST=-$(STRIPPED_PERFLIST) +test-fast: BLACKLIST=-$(STRIPPED_PERFLIST),$(STRIPPED_DRIVERSLIST),$(STRIPPED_DUMPLIST) +test-perf: WHITELIST=$(STRIPPED_PERFLIST) +test-drivers: WHITELIST=$(STRIPPED_DRIVERSLIST) +test-dump: WHITELIST=$(STRIPPED_DUMPLIST) + +test test-basic test-fast test-perf test-drivers test-dump: @mkdir -p $(AUTOTEST_DIR) ; \ cd $(AUTOTEST_DIR) ; \ if [ -f $(RTE_OUTPUT)/app/test ]; then \
Signature · 1 line
2.14.4