Add bits to save/restore URI headrevs

From: Christopher Larson <[email protected]>

This is useful to support BB_NO_NETWORK with AUTOREV, by letting someone ship dumped headrevs to the user, who then inherit this class, which ensures that the cached headrevs are used, and upstream is not contacted at parse time. BB_SRCREV_POLICY will be set to "cache" as well, if it's not already set, as otherwise bitbake will contact upstream to update the cached values.

This is helpful when shipping downloads to someone when there's a desire to support both BB_NO_NETWORK and AUTOREV.

The restore_headrevs.bbclass will restore dumped headrevs at config parse time (add to INHERIT), and the oe.headrevs python module provides a function one can call to dump the headrevs.

The following changes since commit 62bfd1f93f8873611e818c7cc8c13a761d629502:

wireshark: update to version 2.0.0 (2015-11-30 14:39:59 -0500)

are available in the git repository at:

git://github.com/kergoth/meta-openembedded.git headrevs
https://github.com/kergoth/meta-openembedded/tree/headrevs

Christopher Larson (3):
oe: add python namespace package
oe.headrevs: save and restore the bitbake URI headrevs restore_headrevs.bbclass: add

 meta-oe/classes/restore_headrevs.bbclass | 32 +++++++++++++++++++++++++++
 meta-oe/lib/oe/__init__.py               |  2 ++
 meta-oe/lib/oe/headrevs.py               | 38 ++++++++++++++++++++++++++++++++
 3 files changed, 72 insertions(+)
 create mode 100644 meta-oe/classes/restore_headrevs.bbclass
 create mode 100644 meta-oe/lib/oe/__init__.py
 create mode 100644 meta-oe/lib/oe/headrevs.py

2.2.1

From: Christopher Larson <[email protected]>

---
 meta-oe/lib/oe/__init__.py | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 meta-oe/lib/oe/__init__.py

new file mode 100644
index 0000000..3ad9513
@@ -0,0 +1,2 @@
+from pkgutil import extend_path
+__path__ = extend_path(__path__, __name__)

2.2.1

From: Christopher Larson <[email protected]>

This module provides functions to save and restore the BB_URI_HEADREVS from/to the persist_data database. The BB_URI_HEADREVS are the mappings between git refs and the revisions they refer to.

This is useful to fully support BB_NO_NETWORK when AUTOREV is involved, as we can use the dumped mappings rather than contacting upstream at parse time.

---
 meta-oe/lib/oe/headrevs.py | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 meta-oe/lib/oe/headrevs.py

new file mode 100644
index 0000000..9245300
@@ -0,0 +1,38 @@
+"""Save and restore the bitbake URI headrevs.
+
+This module provides functions to save and restore the BB_URI_HEADREVS from/to
+the persist_data database. The BB_URI_HEADREVS are the mappings between git
+refs and the revisions they refer to.
+
+This is useful to fully support BB_NO_NETWORK when AUTOREV is involved, as we
+can use the dumped mappings rather than contacting upstream at parse time. See
+also restore_headrevs.bbclass.
+
+By default, any existing mapping will not be overwritten, as we assume the
+user's mappings are more current than any dumped content.
+"""
+
+
+def copy_persist_domain(d, domain, other_db_path, restore=False, overwrite=False):
+    import contextlib
+
+    source_table = bb.persist_data.persist(domain, d)
+    dest_table = bb.persist_data.SQLTable(other_db_path, domain)
+    if restore:
+        source_table, dest_table = dest_table, source_table
+
+    with contextlib.nested(source_table, dest_table):
+        if overwrite:
+            dest_table.update(source_table)
+        else:
+            for key in source_table:
+                if key not in dest_table:
+                    dest_table[key] = source_table[key]
+
+
+def dump_headrevs(d, dump_db_path):
+    copy_persist_domain(d, 'BB_URI_HEADREVS', dump_db_path)
+
+
+def restore_headrevs(d, dump_db_path):
+    copy_persist_domain(d, 'BB_URI_HEADREVS', dump_db_path, restore=True)

2.2.1

From: Christopher Larson <[email protected]>

If BB_NO_NETWORK=1, this class restores saved bitbake URI headrevs.

This is useful to support BB_NO_NETWORK with AUTOREV, by letting someone ship dumped headrevs to the user, who then inherit this class, which ensures that the cached headrevs are used, and upstream is not contacted at parse time. BB_SRCREV_POLICY will be set to "cache" as well, if it's not already set, as otherwise bitbake will contact upstream to update the cached values.

This only handles restoring dumped headrevs. Using the python module to dump to this database will likely be done at release time, elsewhere.

---
 meta-oe/classes/restore_headrevs.bbclass | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 meta-oe/classes/restore_headrevs.bbclass

new file mode 100644
index 0000000..0d83428
@@ -0,0 +1,32 @@
+# If BB_NO_NETWORK=1, this class restores saved bitbake URI headrevs.
+#
+# This is useful to support BB_NO_NETWORK with AUTOREV, by letting someone
+# ship dumped headrevs to the user, who then inherit this class, which ensures
+# that the cached headrevs are used, and upstream is not contacted at parse
+# time. BB_SRCREV_POLICY will be set to "cache" as well, if it's not already
+# set, as otherwise bitbake will contact upstream to update the cached values.
+#
+# See also lib/oe/headrevs.py.
+#
+# This only handles restoring dumped headrevs. Using the python module to dump
+# to this database will likely be done at release time, elsewhere.
+
+HEADREVS_RESTORE_STAMP ?= "${PERSISTENT_DIR}/headrevs_restored"
+DUMPED_HEADREVS_DB ?= "${COREBASE}/headrevs.db"
+
+python restore_dumped_headrevs() {
+    dump_db_path = d.getVar('DUMPED_HEADREVS_DB', True)
+    if os.path.exists(dump_db_path) and d.getVar('BB_NO_NETWORK', True) == '1':
+        if 'BB_SRCREV_POLICY' not in d:
+            d.setVar('BB_SRCREV_POLICY', 'cache')
+
+        stamp_path = d.getVar('HEADREVS_RESTORE_STAMP', True)
+        if (not os.path.exists(stamp_path) or
+                os.path.getmtime(dump_db_path) > os.path.getmtime(stamp_path)):
+            import oe.headrevs
+            oe.headrevs.restore_headrevs(d, dump_db_path)
+            bb.utils.mkdirhier(os.path.dirname(stamp_path))
+            open(stamp_path, 'w').close()
+}
+restore_dumped_headrevs[eventmask] = "bb.event.ConfigParsed"
+addhandler restore_dumped_headrevs

2.2.1

Maybe I'm missing something, but what's advantage of shipping something with AUTOREV which is then used together with BB_NO_NETWORK?

We already have some implementation in bitbake which dumps latest SRCREVs used by all recipes (including AUTOREV) onces so I would expect that the release will ship this .inc files with locked revisions and if the user of such release wants to use BB_NO_NETWORK he will also include that .inc file with locked revisions.

Is it because it's easier of faster to dump them from the headrefs db you already ship with release and you don't want to generate .inc from buildhistory?

Cheers,

Martin 'JaMa' Jansa     jabber: [email protected]

It's quite useful to not have to go through and manually lock down all your kernels recipes when doing a release. All our releases ship with sources and are BB_NO_NETWORK-capable, as that's a requirement for a lot of folks.

Actually, this was in my patch queue from before buildhistory existed, and I somehow managed to forget about buildhistory-collect-srcrevs :) Feel free to ignore this, in that case.

Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics