support/testing: misc improvements (branch yem/runtime-test-ppd)
cover support/testing: misc improvements (branch yem/runtime-test-ppd)
Hello All!
This small three-patch series brings in two improvements to the runtime testing infrastructure:
- enable top-level parallel build (TLPB) for tests that have
per-package directories (PPD) enabled;
- display the failed commands and its output on assertRunOk().
Finally, it enables PPD, and thus TLPB, for systemd test cases.
Regards,
Yann E. MORIN.
----------------------------------------------------------------
Yann E. MORIN (3):
support/tests: print failed command and output on assertRunOK error support/tests: allow top-level parallel builds support/tests: enable PPD, and thus TLPB, for systemd tests
support/testing/infra/basetest.py | 13 +++++++++---- support/testing/infra/builder.py | 5 ++++- support/testing/tests/init/test_systemd.py | 1 + 3 files changed, 14 insertions(+), 5 deletions(-)
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 1/3 support/tests: print failed command and output on assertRunOK error
Currently, when asserting that a command succeeded, we just capture the return code of the command. If that is not zero, the assertion fails, but the error message is not very splicit:
AssertionError: 1 != 0
Replace the error message with an explicit message that dumps the failed command, the error code, and the resulting output.
--- support/testing/infra/basetest.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
support/testing/infra/basetest.py +8 -3
@@ -88,7 +88,12 @@ class BRTest(BRConfigTest): super(BRTest, self).tearDown() # Run the given 'cmd' with a 'timeout' on the target and - # assert that the command succeeded + # assert that the command succeeded; on error, print the + # faulty command and its output def assertRunOk(self, cmd, timeout=-1): - _, exit_code = self.emulator.run(cmd, timeout) - self.assertEqual(exit_code, 0) + out, exit_code = self.emulator.run(cmd, timeout) + self.assertEqual( + exit_code, + 0, + "\nFailed to run: {}\noutput was:\n{}".format(cmd, ' '+'\n '.join(out)) + )
Signature · 6 lines
2.25.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas Petazzoni via buildroot
AppliedRe: [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error
Quoted 5 lines · On Sat, 24 Dec 2022 10:18:11 +0100 "Yann E. MORIN" <[email protected]>…
Currently, when asserting that a command succeeded, we just capture the return code of the command. If that is not zero, the assertion fails, but the error message is not very splicit:
AssertionError: 1 != 0
Quoted 8 lines
Replace the error message with an explicit message that dumps the failed command, the error code, and the resulting output.
Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>--- support/testing/infra/basetest.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
Applied to master, thanks.
Thomas
Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/3 support/tests: allow top-level parallel builds
In reply to Yann E. MORINRunning tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.
Trigger TLPB when the configuration enables per-package directories.
--- support/testing/infra/basetest.py | 2 +- support/testing/infra/builder.py | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-)
support/testing/infra/basetest.py +1 -1
@@ -52,7 +52,7 @@ class BRConfigTest(unittest.TestCase): def setUp(self): self.show_msg("Starting") - self.b = Builder(self.config, self.builddir, self.logtofile) + self.b = Builder(self.config, self.builddir, self.logtofile, self.jlevel) if not self.keepbuilds: self.b.delete()
support/testing/infra/builder.py +4 -1
@@ -6,11 +6,12 @@ import infra class Builder(object): - def __init__(self, config, builddir, logtofile): + def __init__(self, config, builddir, logtofile, jlevel=None): self.config = '\n'.join([line.lstrip() for line in config.splitlines()]) + '\n' self.builddir = builddir self.logfile = infra.open_log_file(builddir, "build", logtofile) + self.jlevel = jlevel def is_defconfig_valid(self, configfile, defconfig): """Check if the .config is contains all lines present in the defconfig."""
@@ -87,6 +88,8 @@ class Builder(object): env.update(make_extra_env) cmd = ["make", "-C", self.builddir] + if "BR2_PER_PACKAGE_DIRECTORIES=y" in self.config.splitlines() and self.jlevel: + cmd.append(f"-j{self.jlevel}") cmd += make_extra_opts ret = subprocess.call(cmd, stdout=self.logfile, stderr=self.logfile,
Signature · 6 lines
2.25.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds
Quoted 8 lines · On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <[email protected]>…
Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.
Trigger TLPB when the configuration enables per-package directories.
Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>
I certainly support the idea of running some tests with TLPB. However, this implementation makes a confusion between two different settings:
- The existing BRConfigTest.jlevel, which is set by run-tests -j, and used to define BR2_JLEVEL in the Buildroot configuration of each test case. This determines the number of parallel jobs used to build each independent package.
- Your new work, which uses "make -j" to do TLPB... but relies on the same above value, even though it's a completely different setting.
Is this expected?
Best regards,
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds
Thomas, All,
On 2022-12-27 21:45 +0100, Thomas Petazzoni via buildroot spake thusly:
Quoted 19 lines
Quoted 8 lines · On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <[email protected]>…
Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.
Trigger TLPB when the configuration enables per-package directories.
Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>I certainly support the idea of running some tests with TLPB. However, this implementation makes a confusion between two different settings:
- The existing BRConfigTest.jlevel, which is set by run-tests -j, and used to define
BR2_JLEVELin the Buildroot configuration of each test case. This determines the number of parallel jobs used to build each independent package.
Quoted 4 lines
- Your new work, which uses "make -j" to do TLPB... but relies on the same above value, even though it's a completely different setting.
Is this expected?
Yes, this is the intended behaviour, which I was explicitly seeking.
So, if one runs with PPD and TLPB (outside the run-time infra), one would do something like:
$ make -jN
This spawns a top-level make process that is parallel. In turn, in rules, when we call $(MAKE), this is the magic that tells make that it is recursive, but that it should use the jobserver from the calling process.
So, in this case, the BR2_JLEVEL is unused by whatever uses the make jobserver; only the number of jobs in the top-level jobserver is meaningful, i.e. whatever we pass as -jN.
The exception, of course, is whatever uses BR2_JLEVEL but does not talk to the jobserver, but this is mostly a few packages (scons, waf et al.). Even ninja packages do talk to the top-level jobserver, now that we use the ninja fork that knows to talk to it.
So, yes, using top-level -jN with the same value as BR2_JLEVEL is exactly what I intended to do.
Unless I totally missed something...
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds
Thomas, All,
On 2022-12-27 21:53 +0100, Yann E. MORIN spake thusly:
Quoted 32 lines
On 2022-12-27 21:45 +0100, Thomas Petazzoni via buildroot spake thusly:
Quoted 19 lines
Quoted 8 lines · On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <[email protected]>…
Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.
Trigger TLPB when the configuration enables per-package directories.
Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>I certainly support the idea of running some tests with TLPB. However, this implementation makes a confusion between two different settings:
- The existing BRConfigTest.jlevel, which is set by run-tests -j, and used to define
BR2_JLEVELin the Buildroot configuration of each test case. This determines the number of parallel jobs used to build each independent package.Quoted 4 lines
- Your new work, which uses "make -j" to do TLPB... but relies on the same above value, even though it's a completely different setting.
Is this expected?
Yes, this is the intended behaviour, which I was explicitly seeking.
So, if one runs with PPD and TLPB (outside the run-time infra), one would do something like:
$ make -jN
Quoted 8 lines
This spawns a top-level make process that is parallel. In turn, in rules, when we call $(MAKE), this is the magic that tells make that it is recursive, but that it should use the jobserver from the calling process.
So, in this case, the
BR2_JLEVELis unused by whatever uses the make jobserver; only the number of jobs in the top-level jobserver is meaningful, i.e. whatever we pass as -jN.
Slight correction: BR2_JLEVEL does have an actual effect, but only if its value is lower than the one we pass as -jN.
So, in practice, we want to have BR2_JLEVEL equal to the top-level -jN.
Regards,
Yann E. MORIN.
Quoted 24 lines
The exception, of course, is whatever uses
BR2_JLEVELbut does not talk to the jobserver, but this is mostly a few packages (scons, waf et al.). Even ninja packages do talk to the top-level jobserver, now that we use the ninja fork that knows to talk to it.So, yes, using top-level -jN with the same value as
BR2_JLEVELis exactly what I intended to do.Unless I totally missed something...
Regards,
Yann E. MORIN.Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds
In reply to Yann E. MORINQuoted 4 lines · On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <[email protected]>…
Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.
Quoted 8 lines
Trigger TLPB when the configuration enables per-package directories.
Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>--- support/testing/infra/basetest.py | 2 +- support/testing/infra/builder.py | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-)
Thanks to the additional explanation you provided, I understood better the idea, and therefore applied the patch. I updated the commit log a little bit to summarize the explanation you gave, and applied. Thanks!
Thomas
Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 3/3 support/tests: enable PPD, and thus TLPB, for systemd tests
In reply to Yann E. MORIN--- support/testing/tests/init/test_systemd.py | 1 + 1 file changed, 1 insertion(+)
support/testing/tests/init/test_systemd.py +1 -0
@@ -13,6 +13,7 @@ class InitSystemSystemdBase(InitSystemBase): BR2_INIT_SYSTEMD=y BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0" # BR2_TARGET_ROOTFS_TAR is not set + BR2_PER_PACKAGE_DIRECTORIES=y """ def check_systemd(self, fs):
Signature · 6 lines
2.25.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas Petazzoni via buildroot
AppliedRe: [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests
Quoted 5 lines · On Sat, 24 Dec 2022 10:18:13 +0100 "Yann E. MORIN" <[email protected]>…
Signed-off-by Yann E. MORIN <[email protected]>Cc Thomas Petazzoni <[email protected]>--- support/testing/tests/init/test_systemd.py | 1 + 1 file changed, 1 insertion(+)
Applied to master, thanks.
Thomas
Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot