Skip to contentSkip to conversation

bb: Add prefuncs and postfuncs ordering

PATCH In review Joshua Watt 2 participants 3 messages Jun 26 – Jun 28 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bb: Add prefuncs and postfuncs ordering

Adds the ability for functions specified in prefuncs and postfuncs to specify the order in which they must execute. This allows specific functions to indicate that they must run first or last (or second, second to last, etc.) without needing all places in the code that manipulate the function list to have to agree on how to order them (a good example of this is sstate.bbclass and buildhistory.bbclass).

Ordering is done similar to how python list indices work; 0 is always sorted first, 1 is second, -2 is second to last, -1 is last, etc. In the event of functions having duplicate values, they are grouped together, but keep their same relative ordering. If one or more functions do not have an order assigned, they executed between the positive order functions and the negative order functions, and keep their same relative order (this means that existing behavior is preserved in the absence of any ordering information).

---
 .../bitbake-user-manual-metadata.rst          | 31 ++++++++++-
 lib/bb/build.py                               | 52 ++++++++++++++++++-
 lib/bb/tests/runqueue-tests/recipes/funcs.inc | 44 ++++++++++++++++
 .../tests/runqueue-tests/recipes/postfuncs.bb | 10 ++++
 .../tests/runqueue-tests/recipes/prefuncs.bb  | 10 ++++
 lib/bb/tests/runqueue.py                      | 26 +++++++++-
 6 files changed, 168 insertions(+), 5 deletions(-)
 create mode 100644 lib/bb/tests/runqueue-tests/recipes/funcs.inc
 create mode 100644 lib/bb/tests/runqueue-tests/recipes/postfuncs.bb
 create mode 100644 lib/bb/tests/runqueue-tests/recipes/prefuncs.bb
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +29 -2
index 0c7c3ff99..607f98d02 100644
@@ -1827,10 +1827,37 @@ functionality of the task:
          the :term:`BB_NUMBER_THREADS` variable causes ``number_threads`` to
          have no effect.
 
--  ``[postfuncs]``: List of functions to call after the completion of
-   the task.
+-  ``[postfuncs]``: List of functions to call after the completion of the task.
+   Functions specified may set a ``[postfunc-order]`` flag to specify the order
+   in which they will execute. The ordering is based on the way that python
+   does list indexes. The first group of functions executed are those with a
+   positive or zero ``[postfunc-order]`` flag, in increasing order (i.e.  "0"
+   is first, "1" is second, etc.). The next group executed are functions that
+   do not have a ``[postfunc-order]`` flag set, in the relative order they are
+   specified in ``[postfuncs]``. The last group are functions with a negative
+   ``[postfunc-order]`` flag, in increasing order (i.e. "-2" is the second to
+   last, "-1" is the last). If two functions have the same ``[postfunc-order]``
+   value, they will be grouped together, but keep the same relative order they
+   had in ``[postfuncs]``.
+
+-  ``[postfunc-order]``: The order this function will be executed in when
+   listed in ``[postfuncs]``.
 
 -  ``[prefuncs]``: List of functions to call before the task executes.
+   Functions specified may set a ``[prefunc-order]`` flag to specify the order
+   in which they will execute.  The ordering is based on the way that python
+   does list indexes. The first group of functions executed are those with a
+   positive or zero ``[prefunc-order]`` flag, in increasing order (i.e.  "0" is
+   first, "1" is second, etc.). The next group executed are functions that do
+   not have a ``[prefunc-order]`` flag set, in the relative order they are
+   specified in ``[prefuncs]``. The last group are functions with a negative
+   ``[prefunc-order]`` flag, in increasing order (i.e. "-2" is the second to
+   last, "-1" is the last). If two functions have the same ``[prefunc-order]``
+   value, they will be grouped together, but keep the same relative order they
+   had in ``[prefuncs]``.
+
+-  ``[prefunc-order]``: The order this function will be executed in when
+   listed in ``[prefuncs]``.
 
 -  ``[rdepends]``: Controls inter-task runtime dependencies. See the
    :term:`RDEPENDS` variable, the
lib/bb/build.py +50 -2

66 lines not on this page · raw message

lib/bb/tests/runqueue-tests/recipes/funcs.inc new file +44 -0

44 lines not on this page · raw message

lib/bb/tests/runqueue-tests/recipes/postfuncs.bb new file +10 -0
new file mode 100644
index 000000000..be82b7302
@@ -0,0 +1,10 @@
+require funcs.inc
+
+do_build[postfuncs] += "func_8 func_6 func_7 func_4 func_5 func_2 func_3 func_1"
+
+func_1[postfunc-order] = "0"
+func_2[postfunc-order] = "1"
+func_3[postfunc-order] = "1"
+func_6[postfunc-order] = "-3"
+func_7[postfunc-order] = "-3"
+func_8[postfunc-order] = "-2"
lib/bb/tests/runqueue-tests/recipes/prefuncs.bb new file +10 -0
new file mode 100644
index 000000000..6c8d1e380
@@ -0,0 +1,10 @@
+require funcs.inc
+
+do_build[prefuncs] += "func_8 func_6 func_7 func_4 func_5 func_2 func_3 func_1"
+
+func_1[prefunc-order] = "0"
+func_2[prefunc-order] = "1"
+func_3[prefunc-order] = "1"
+func_6[prefunc-order] = "-3"
+func_7[prefunc-order] = "-3"
+func_8[prefunc-order] = "-2"
lib/bb/tests/runqueue.py +25 -1
index 74f5ded2e..ecc6ab6db 100644
@@ -12,6 +12,7 @@ import tempfile
 import subprocess
 import sys
 import time
+from contextlib import contextmanager
 
 #
 # TODO:
@@ -53,6 +54,14 @@ class RunQueueTests(unittest.TestCase):
                 os.remove(tasklog)
         return tasks
 
+    @contextmanager
+    def tempdir(self):
+        with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+            try:
+                yield tempdir
+            finally:
+                self.shutdown(tempdir)
+
     def test_no_setscenevalid(self):
         with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
             cmd = ["bitbake", "a1"]
@@ -403,8 +412,23 @@ class RunQueueTests(unittest.TestCase):
 
             self.shutdown(tempdir)
 
+    def test_postfuncs(self):
+        with self.tempdir() as tempdir:
+            log = f"{tempdir}/postlog.txt"
+            tasks = self.run_bitbakecmd(["bitbake", "postfuncs"], tempdir, extraenv={"ORDER_LOG": log})
+            self.assertTrue(os.path.exists(log), "Log file not written")
+            with open(log, "r") as f:
+                self.assertEqual(f.read(), " build 1 2 3 4 5 6 7 8")
+
+    def test_prefuncs(self):
+        with self.tempdir() as tempdir:
+            log = f"{tempdir}/prelog.txt"
+            tasks = self.run_bitbakecmd(["bitbake", "prefuncs"], tempdir, extraenv={"ORDER_LOG": log})
+            self.assertTrue(os.path.exists(log), "Log file not written")
+            with open(log, "r") as f:
+                self.assertEqual(f.read(), " 1 2 3 4 5 6 7 8 build")
+
     def shutdown(self, tempdir):
         # Wait for the hashserve socket to disappear else we'll see races with the tempdir cleanup
         while (os.path.exists(tempdir + "/hashserve.sock") or os.path.exists(tempdir + "cache/hashserv.db-wal") or os.path.exists(tempdir + "/bitbake.lock")):
             time.sleep(0.5)
-
Signature · 1 line
2.54.0
#1 lore raw
Quoted 16 lines · Joshua Watt via lists.openembedded.org

Adds the ability for functions specified in prefuncs and postfuncs to specify the order in which they must execute. This allows specific functions to indicate that they must run first or last (or second, second to last, etc.) without needing all places in the code that manipulate the function list to have to agree on how to order them (a good example of this is sstate.bbclass and buildhistory.bbclass).

Ordering is done similar to how python list indices work; 0 is always sorted first, 1 is second, -2 is second to last, -1 is last, etc. In the event of functions having duplicate values, they are grouped together, but keep their same relative ordering. If one or more functions do not have an order assigned, they executed between the positive order functions and the negative order functions, and keep their same relative order (this means that existing behavior is preserved in the absence of any ordering information).

Quoted 1 line

I've wondered about something like this for a long time, the trouble is it does not scale well and I've always concluded it wouldn't really help overall and solve the real problem.

It solves an immediate issue of making one thing run first, or last. If a second comes along, it can just about handle that. The trouble is when you have three or more and they all want to be "last", or "first", you can't define that in the metadata. This just gets worse when you think of independent layers trying to pick numeric values. This is already a pain with layer priorities and I wish we'd never used numbers.

You can already get some idea of ordering just by the += and =+ operations ordering the data, or reordering the variable contents using anon python.

You could even define a filter function for the variable which could reorder things with a custom order function.

So I'm probably against adding this level of complexity when we already have enough of it and it doesn't fully solve the issue anyway.

If we did something, I'd probably look in the direction of "after" and "before" operations like tasks, trying to avoid some of the issues we have there.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 38 lines · Richard Purdie via lists.openembedded.org
Quoted 16 lines · Joshua Watt via lists.openembedded.org

Adds the ability for functions specified in prefuncs and postfuncs to specify the order in which they must execute. This allows specific functions to indicate that they must run first or last (or second, second to last, etc.) without needing all places in the code that manipulate the function list to have to agree on how to order them (a good example of this is sstate.bbclass and buildhistory.bbclass).

Ordering is done similar to how python list indices work; 0 is always sorted first, 1 is second, -2 is second to last, -1 is last, etc. In the event of functions having duplicate values, they are grouped together, but keep their same relative ordering. If one or more functions do not have an order assigned, they executed between the positive order functions and the negative order functions, and keep their same relative order (this means that existing behavior is preserved in the absence of any ordering information).

Quoted 1 line

I've wondered about something like this for a long time, the trouble is it does not scale well and I've always concluded it wouldn't really help overall and solve the real problem.

It solves an immediate issue of making one thing run first, or last. If a second comes along, it can just about handle that. The trouble is when you have three or more and they all want to be "last", or "first", you can't define that in the metadata. This just gets worse when you think of independent layers trying to pick numeric values. This is already a pain with layer priorities and I wish we'd never used numbers.

You can already get some idea of ordering just by the += and =+ operations ordering the data, or reordering the variable contents using anon python.

You could even define a filter function for the variable which could reorder things with a custom order function.

Quoted 6 lines

So I'm probably against adding this level of complexity when we already have enough of it and it doesn't fully solve the issue anyway.

If we did something, I'd probably look in the direction of "after" and "before" operations like tasks, trying to avoid some of the issues we have there.

I couldn't help thinking about this more, the syntax I was wondering about was something along the lines of:

do_xxx[postfuncs] += "functionC[after-functionB,after-functionA,before-functionD]"

which is horrible, but at least explicit.

Cheers,

Richard