Skip to contentSkip to conversation

runqueue: add 'default' mc translation to mcdepends

PATCH In review Mark Hatle 3 participants 5 messages Jan 24, 2025 – Jan 26, 2025

lore
Share & tools
lore

Mark Hatle

author
#0 lore raw

PATCH 1/2 runqueue: add 'default' mc translation to mcdepends

From: Mark Hatle <[email protected]>

The default mc is '' (blank), however BB_CURRENT_MC returns 'default'. Allow a user to specify a mcdepend such as:

  mc:${BB_CURRENT_MC}:m2:recipe:task

Both '' (blank) and 'default' are now supported and have equivalent meaning.

---
 bitbake/lib/bb/runqueue.py                        | 4 ++++
 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb | 2 ++
 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb | 0
 bitbake/lib/bb/tests/runqueue-tests/recipes/i1.bb | 0
 bitbake/lib/bb/tests/runqueue.py                  | 7 +++++++
 5 files changed, 13 insertions(+)
 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb
 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb
 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/i1.bb
bitbake/lib/bb/runqueue.py +4 -0
index ffb2d28494..f941749b0d 100644
@@ -723,7 +723,11 @@ class RunQueueData:
                 mcdependency = dep.split(':')
                 pn = mcdependency[3]
                 frommc = mcdependency[1]
+                if frommc == 'default':
+                    frommc = ''
                 mcdep = mcdependency[2]
+                if mcdep == 'default':
+                    mcdep = ''
                 deptask = mcdependency[4]
                 if mcdep not in taskData:
                     bb.fatal("Multiconfig '%s' is referenced in multiconfig dependency '%s' but not enabled in BBMULTICONFIG?" % (mcdep, dep))
bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb new file +2 -0
new file mode 100644
index 0000000000..20ce8dd3ad
@@ -0,0 +1,2 @@
+do_build[mcdepends] = "mc::mc-1:h1:do_build mc:default:mc_2:i1:do_build"
+
bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
bitbake/lib/bb/tests/runqueue-tests/recipes/i1.bb new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
bitbake/lib/bb/tests/runqueue.py +7 -0
index cc87e8d6a8..cea2ca13fe 100644
@@ -314,6 +314,13 @@ class RunQueueTests(unittest.TestCase):
                        ["mc_2:a1:%s" % t for t in rerun_tasks]
             self.assertEqual(set(tasks), set(expected))
 
+            # Test that mc::... and mc:default:... both work
+            tasks = self.run_bitbakecmd(["bitbake", "g1"], tempdir, "", extraenv=extraenv, cleanup=True)
+            expected = ["g1:%s" % t for t in self.alltasks] + \
+                       ["mc-1:h1:%s" % t for t in self.alltasks] + \
+                       ["mc_2:i1:%s" % t for t in self.alltasks]
+            self.assertEqual(set(tasks), set(expected))
+
             self.shutdown(tempdir)
 
     def test_hashserv_single(self):
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 2/2 runqueue: Verify mcdepends are valid

From: Mark Hatle <[email protected]>

In order to avoid a potentially confusing backtrace, check that the mcdepend is valid when we add it.

Add a test case to ensure invalid configurations are caught and trigger an error.

---
 bitbake/lib/bb/runqueue.py                        | 2 ++
 bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb | 2 ++
 bitbake/lib/bb/tests/runqueue.py                  | 9 +++++++++
 3 files changed, 13 insertions(+)
 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb
bitbake/lib/bb/runqueue.py +2 -0
index f941749b0d..90f606469b 100644
@@ -734,6 +734,8 @@ class RunQueueData:
                 if mc == frommc:
                     fn = taskData[mcdep].build_targets[pn][0]
                     newdep = '%s:%s' % (fn,deptask)
+                    if newdep not in taskData[mcdep].taskentries:
+                        bb.fatal("Task mcdepends on non-existent task %s" % (newdep))
                     taskData[mc].taskentries[tid].tdepends.append(newdep)
 
         for mc in taskData:
bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb new file +2 -0
new file mode 100644
index 0000000000..3c7dca0257
@@ -0,0 +1,2 @@
+do_build[mcdepends] = "mc::mc-1:h1:do_invalid"
+
bitbake/lib/bb/tests/runqueue.py +9 -0
index cea2ca13fe..4662efbf55 100644
@@ -321,6 +321,15 @@ class RunQueueTests(unittest.TestCase):
                        ["mc_2:i1:%s" % t for t in self.alltasks]
             self.assertEqual(set(tasks), set(expected))
 
+            # Check what happens with an invalid task depednency
+            passed = False
+            try:
+                self.run_bitbakecmd(["bitbake", "j1"], tempdir, "", extraenv=extraenv, cleanup=True)
+            except AssertionError as e:
+                print("Expected failure: %s" % e)
+                passed = True
+            self.assertEqual(True, passed)
+
             self.shutdown(tempdir)
 
     def test_hashserv_single(self):
Signature · 1 line
2.34.1
#3 lore raw
In reply to Mark Hatle
Quoted 5 lines · Mark Hatle via lists.openembedded.org

From: Mark Hatle <[email protected]>

The default mc is '' (blank), however BB_CURRENT_MC returns 'default'. Allow a user to specify a mcdepend such as:

Quoted 7 lines

  mc:${BB_CURRENT_MC}:m2:recipe:task

Both '' (blank) and 'default' are now supported and have equivalent meaning.

I'm wondering if we should remove "default" entirely, it is confusing and I'm not sure we want to complicate the code with this. I'm going to test a patch doing that.

Cheers,

Richard

#2 lore raw

Re: [bitbake-devel] [PATCH 2/2] runqueue: Verify mcdepends are valid

In reply to Mark Hatle
Quoted 50 lines · Mark Hatle via lists.openembedded.org

From: Mark Hatle <[email protected]>

In order to avoid a potentially confusing backtrace, check that the mcdepend is valid when we add it.

Add a test case to ensure invalid configurations are caught and trigger an error.

---
 bitbake/lib/bb/runqueue.py                        | 2 ++ bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb | 2 ++ bitbake/lib/bb/tests/runqueue.py                  | 9 +++++++++

 3 files changed, 13 insertions(+)

 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb

bitbake/lib/bb/runqueue.py +0 -0
index f941749b0d..90f606469b 100644
@@ -734,6 +734,8 @@ class RunQueueData:

                 if mc == frommc:
                     fn = taskData[mcdep].build_targets[pn][0]
                     newdep = '%s:%s' % (fn,deptask)
+                    if newdep not in taskData[mcdep].taskentries:
+                        bb.fatal("Task mcdepends on non-existent task %s" % (newdep)) taskData[mc].taskentries[tid].tdepends.append(newdep)
         for mc in taskData:

bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb new file +2 -0
new file mode 100644
index 0000000000..3c7dca0257
@@ -0,0 +1,2 @@
+do_build[mcdepends] = "mc::mc-1:h1:do_invalid"
+
bitbake/lib/bb/tests/runqueue.py +0 -0
index cea2ca13fe..4662efbf55 100644
@@ -321,6 +321,15 @@ class RunQueueTests(unittest.TestCase):

                        ["mc_2:i1:%s" % t for t in self.alltasks] self.assertEqual(set(tasks), set(expected))

+            # Check what happens with an invalid task depednency
+            passed = False
+            try:
+                self.run_bitbakecmd(["bitbake", "j1"], tempdir, "", extraenv=extraenv, cleanup=True)

Quoted 8 lines

+            except AssertionError as e:
+                print("Expected failure: %s" % e) +                passed = True
+            self.assertEqual(True, passed) +
             self.shutdown(tempdir)

     def test_hashserv_single(self):

Firstly, I *really* appreciate people adding tests, its great to see so thanks for that.

The test needs a few tweaks, just to follow best practice.

Basically you can use assertRaises() to test it raises an exception and we should really check that it says "Command.*failed" and "mcdepends on non-existent task.*do_invalid".

https://www.geeksforgeeks.org/test-if-a-function-throws-an-exception-in-python/

Another option would be to have a new parameter to run_bitbakecmd which just makes it return the failure output.

Cheers,

Richard

#4 lore raw
In reply to Mark Hatle
Quoted 5 lines · Mark Hatle

From: Mark Hatle <[email protected]>

The default mc is '' (blank), however BB_CURRENT_MC returns 'default'. Allow a user to specify a mcdepend such as:

Quoted 8 lines
  mc:${BB_CURRENT_MC}:m2:recipe:task

Both '' (blank) and 'default' are now supported and have equivalent meaning.

---

Hi Mark,

Thanks for your patch.

I believe one of the patch from this series is responsible for some failures we can see on the autobuilder:

FAIL: test_multiconfig_mcdepends (bb.tests.runqueue.RunQueueTests.test_multiconfig_mcdepends)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/tests/runqueue.py", line 41, in run_bitbakecmd
output = subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bitbake', 'g1']' returned non-zero exit status 1.

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/846/steps/11/logs/stdio

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com