Skip to contentSkip to conversation

support/testing: misc improvements (branch yem/runtime-test-ppd)

PATCH Applied Yann E. MORIN 2 participants 10 messages Dec 24, 2022 – Feb 7, 2023

lore
Share & tools
lore
#0 lore raw

cover support/testing: misc improvements (branch yem/runtime-test-ppd)

Hello All!

This small three-patch series brings in two improvements to the runtime testing infrastructure:
- enable top-level parallel build (TLPB) for tests that have

    per-package directories (PPD) enabled;

- display the failed commands and its output on assertRunOk().

Finally, it enables PPD, and thus TLPB, for systemd test cases.

Regards,
Yann E. MORIN.

----------------------------------------------------------------
Yann E. MORIN (3):
support/tests: print failed command and output on assertRunOK error support/tests: allow top-level parallel builds support/tests: enable PPD, and thus TLPB, for systemd tests

 support/testing/infra/basetest.py          | 13 +++++++++----
 support/testing/infra/builder.py           |  5 ++++-
 support/testing/tests/init/test_systemd.py |  1 +
 3 files changed, 14 insertions(+), 5 deletions(-)
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

PATCH 1/3 support/tests: print failed command and output on assertRunOK error

Currently, when asserting that a command succeeded, we just capture the return code of the command. If that is not zero, the assertion fails, but the error message is not very splicit:

    AssertionError: 1 != 0

Replace the error message with an explicit message that dumps the failed command, the error code, and the resulting output.

Cc Thomas Petazzoni <[email protected]>
---
 support/testing/infra/basetest.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
support/testing/infra/basetest.py +8 -3
index 96c6848dfc..45bcd4c2e2 100644
@@ -88,7 +88,12 @@ class BRTest(BRConfigTest):
         super(BRTest, self).tearDown()
 
     # Run the given 'cmd' with a 'timeout' on the target and
-    # assert that the command succeeded
+    # assert that the command succeeded; on error, print the
+    # faulty command and its output
     def assertRunOk(self, cmd, timeout=-1):
-        _, exit_code = self.emulator.run(cmd, timeout)
-        self.assertEqual(exit_code, 0)
+        out, exit_code = self.emulator.run(cmd, timeout)
+        self.assertEqual(
+            exit_code,
+            0,
+            "\nFailed to run: {}\noutput was:\n{}".format(cmd, '  '+'\n  '.join(out))
+        )
Signature · 6 lines
2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Re: [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error

Quoted 5 lines · On Sat, 24 Dec 2022 10:18:11 +0100 "Yann E. MORIN" <[email protected]>…

Currently, when asserting that a command succeeded, we just capture the return code of the command. If that is not zero, the assertion fails, but the error message is not very splicit:

    AssertionError: 1 != 0
Quoted 8 lines

Replace the error message with an explicit message that dumps the failed command, the error code, and the resulting output.

Cc Thomas Petazzoni <[email protected]>
---
 support/testing/infra/basetest.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

Applied to master, thanks.

Thomas

Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

PATCH 2/3 support/tests: allow top-level parallel builds

In reply to Yann E. MORIN

Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.

Trigger TLPB when the configuration enables per-package directories.

Cc Thomas Petazzoni <[email protected]>
---
 support/testing/infra/basetest.py | 2 +-
 support/testing/infra/builder.py  | 5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)
support/testing/infra/basetest.py +1 -1
index 45bcd4c2e2..670c7213d6 100644
@@ -52,7 +52,7 @@ class BRConfigTest(unittest.TestCase):
 
     def setUp(self):
         self.show_msg("Starting")
-        self.b = Builder(self.config, self.builddir, self.logtofile)
+        self.b = Builder(self.config, self.builddir, self.logtofile, self.jlevel)
 
         if not self.keepbuilds:
             self.b.delete()
support/testing/infra/builder.py +4 -1
index 922a707220..a2abb9ed89 100644
@@ -6,11 +6,12 @@ import infra
 
 
 class Builder(object):
-    def __init__(self, config, builddir, logtofile):
+    def __init__(self, config, builddir, logtofile, jlevel=None):
         self.config = '\n'.join([line.lstrip() for line in
                                  config.splitlines()]) + '\n'
         self.builddir = builddir
         self.logfile = infra.open_log_file(builddir, "build", logtofile)
+        self.jlevel = jlevel
 
     def is_defconfig_valid(self, configfile, defconfig):
         """Check if the .config is contains all lines present in the defconfig."""
@@ -87,6 +88,8 @@ class Builder(object):
         env.update(make_extra_env)
 
         cmd = ["make", "-C", self.builddir]
+        if "BR2_PER_PACKAGE_DIRECTORIES=y" in self.config.splitlines() and self.jlevel:
+            cmd.append(f"-j{self.jlevel}")
         cmd += make_extra_opts
 
         ret = subprocess.call(cmd, stdout=self.logfile, stderr=self.logfile,
Signature · 6 lines
2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw

Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

Quoted 8 lines · On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <[email protected]>…

Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.

Trigger TLPB when the configuration enables per-package directories.

Cc Thomas Petazzoni <[email protected]>

I certainly support the idea of running some tests with TLPB. However, this implementation makes a confusion between two different settings:

- The existing BRConfigTest.jlevel, which is set by run-tests -j, and used to define BR2_JLEVEL in the Buildroot configuration of each test case. This determines the number of parallel jobs used to build each independent package.

- Your new work, which uses "make -j" to do TLPB... but relies on the same above value, even though it's a completely different setting.

Is this expected?

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

Thomas, All,

On 2022-12-27 21:45 +0100, Thomas Petazzoni via buildroot spake thusly:

Quoted 19 lines
Quoted 8 lines · On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <[email protected]>…

Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.

Trigger TLPB when the configuration enables per-package directories.

Cc Thomas Petazzoni <[email protected]>

I certainly support the idea of running some tests with TLPB. However, this implementation makes a confusion between two different settings:

- The existing BRConfigTest.jlevel, which is set by run-tests -j, and used to define BR2_JLEVEL in the Buildroot configuration of each test case. This determines the number of parallel jobs used to build each independent package.

Quoted 4 lines

- Your new work, which uses "make -j" to do TLPB... but relies on the same above value, even though it's a completely different setting.

Is this expected?

Yes, this is the intended behaviour, which I was explicitly seeking.

So, if one runs with PPD and TLPB (outside the run-time infra), one would do something like:

    $ make -jN

This spawns a top-level make process that is parallel. In turn, in rules, when we call $(MAKE), this is the magic that tells make that it is recursive, but that it should use the jobserver from the calling process.

So, in this case, the BR2_JLEVEL is unused by whatever uses the make jobserver; only the number of jobs in the top-level jobserver is meaningful, i.e. whatever we pass as -jN.

The exception, of course, is whatever uses BR2_JLEVEL but does not talk to the jobserver, but this is mostly a few packages (scons, waf et al.). Even ninja packages do talk to the top-level jobserver, now that we use the ninja fork that knows to talk to it.

So, yes, using top-level -jN with the same value as BR2_JLEVEL is exactly what I intended to do.

Unless I totally missed something...

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#6 lore raw

Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

Thomas, All,

On 2022-12-27 21:53 +0100, Yann E. MORIN spake thusly:

Quoted 32 lines

On 2022-12-27 21:45 +0100, Thomas Petazzoni via buildroot spake thusly:

Quoted 19 lines
Quoted 8 lines · On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <[email protected]>…

Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.

Trigger TLPB when the configuration enables per-package directories.

Cc Thomas Petazzoni <[email protected]>

I certainly support the idea of running some tests with TLPB. However, this implementation makes a confusion between two different settings:

- The existing BRConfigTest.jlevel, which is set by run-tests -j, and used to define BR2_JLEVEL in the Buildroot configuration of each test case. This determines the number of parallel jobs used to build each independent package.

Quoted 4 lines

- Your new work, which uses "make -j" to do TLPB... but relies on the same above value, even though it's a completely different setting.

Is this expected?

Yes, this is the intended behaviour, which I was explicitly seeking.

So, if one runs with PPD and TLPB (outside the run-time infra), one would do something like:

    $ make -jN
Quoted 8 lines

This spawns a top-level make process that is parallel. In turn, in rules, when we call $(MAKE), this is the magic that tells make that it is recursive, but that it should use the jobserver from the calling process.

So, in this case, the BR2_JLEVEL is unused by whatever uses the make jobserver; only the number of jobs in the top-level jobserver is meaningful, i.e. whatever we pass as -jN.

Slight correction: BR2_JLEVEL does have an actual effect, but only if its value is lower than the one we pass as -jN.

So, in practice, we want to have BR2_JLEVEL equal to the top-level -jN.

Regards,
Yann E. MORIN.

Quoted 24 lines

The exception, of course, is whatever uses BR2_JLEVEL but does not talk to the jobserver, but this is mostly a few packages (scons, waf et al.). Even ninja packages do talk to the top-level jobserver, now that we use the ninja fork that knows to talk to it.

So, yes, using top-level -jN with the same value as BR2_JLEVEL is exactly what I intended to do.

Unless I totally missed something...

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#7 lore raw

Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

In reply to Yann E. MORIN
Quoted 4 lines · On Sat, 24 Dec 2022 10:18:12 +0100 "Yann E. MORIN" <[email protected]>…

Running tests with top-level parallel builds can speed up running some tests, expecially those that have a lot of packages like the systemd init tests.

Quoted 8 lines

Trigger TLPB when the configuration enables per-package directories.

Cc Thomas Petazzoni <[email protected]>
---
 support/testing/infra/basetest.py | 2 +-
 support/testing/infra/builder.py  | 5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)

Thanks to the additional explanation you provided, I understood better the idea, and therefore applied the patch. I updated the commit log a little bit to summarize the explanation you gave, and applied. Thanks!

Thomas

Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#8 lore raw

PATCH 3/3 support/tests: enable PPD, and thus TLPB, for systemd tests

In reply to Yann E. MORIN
Cc Thomas Petazzoni <[email protected]>
---
 support/testing/tests/init/test_systemd.py | 1 +
 1 file changed, 1 insertion(+)
support/testing/tests/init/test_systemd.py +1 -0
index 06c42736ae..ddc32b0838 100644
@@ -13,6 +13,7 @@ class InitSystemSystemdBase(InitSystemBase):
         BR2_INIT_SYSTEMD=y
         BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
         # BR2_TARGET_ROOTFS_TAR is not set
+        BR2_PER_PACKAGE_DIRECTORIES=y
         """
 
     def check_systemd(self, fs):
Signature · 6 lines
2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#9 lore raw

Re: [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests

Quoted 5 lines · On Sat, 24 Dec 2022 10:18:13 +0100 "Yann E. MORIN" <[email protected]>…
Cc Thomas Petazzoni <[email protected]>
---
 support/testing/tests/init/test_systemd.py | 1 +
 1 file changed, 1 insertion(+)

Applied to master, thanks.

Thomas

Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot